Defenders: Caller based EC2 security

The risk with security credentials is that they get exposed an are being used elsewhere. What if we could prevent that the are being used elsewhere. The idea from the article of William Bengston from netflix was: Dynamically locking credentials to the environment.

This implementation of this idea is much more simple with the cdk. So, let’s defend ourselves!

Our story here is the battle of the defenders (tm). They are DC comix superheroes and they have some special attributes - more on this later…

But first we need a battleground. We will code the Resources with the CDK in node style. More on the cdk in the blog

Battleground

We build a vpc as battleground. As a first line of defense, we want to secure our battleground. So a security group is included, which first determines the current public ip of our workstation and limit ssh access to the EC2 instances to this public ip.

First line of defense - Get rid of ‘0.0.0.0/0’

The public IP from our workstation is read via AWS Checkip and the put into a SecurityGroup:

GetLocalIp

import * as request from "request-promise-native";

export async function GetLocalIp() : Promise<string> {
    var clientIp: string;
    const baseUrl = 'http://checkip.amazonaws.com/';

    var options = {
        uri: baseUrl,
    };

    var result = await request.get(options);
    if(result.indexOf(",") > -1) {
        var arr = result.split(",");
        result = arr[1]

    }
    clientIp = result.trim()+"/32";

    return clientIp;

}

In this node code snippet we make a simple request to the checkip site from aws. Sometimes you get two ips (don’t ask me why). In this case we drop the first one.

Battelground.ts

const defVpc = new Vpc(this, 'defendersVPC', {});
this.vpc = defVpc;
    
const clientIp = GetLocalIp();

clientIp.then((ip) => {

    const sg = new SecurityGroup(this, "defenderDemo",{
        vpc: defVpc,
        securityGroupName: "SSH incoming",
        description:  "SSH Incoming on current public ip",
        allowAllOutbound: true,
    });
    Tag.add(sg, "Name", "dynamicIncomingSSHClient");

    sg.addIngressRule(Peer.ipv4(ip), Port.tcp(22), "Ssh incoming")

    this.sshIncomingSecurityGroup = sg;
})

In this code snippet we create a one-line-vpc (and I love the one-line-vpc! ) and attach the security group to it.

In creating this SecurityGroup for my work from a mobile office we madee sure that there is no 0.0.0.0/0 - SSH-open-to-the-world incoming line in the security group.

With this being done, we create our first superhero instance: Meet Matt.

Matt with limited security superpower and second line of defense

There is “the DareDevil” Matt (oops, now you know his secret identity). Matt has some limited superpowers, so on the screen he mostly gets serious damage.

Matt is the codename for our mildly secured credentials. Let’s build Matt:

Use limited ssh access security group of the battleground

    const mattSGId = Fn.importValue('dynamicIncomingSSHClient');
    const mattSG = SecurityGroup.fromSecurityGroupId(this, "Matt Security Group", mattSGId);

Battleground exports the security group id as dynamicIncomingSSHClient - we import the id and use the security group.

Matt gets some rights

Matt now gets some rights, with the AWS managed policy AmazonEC2ReadOnlyAccess:

  const instanceRole = new Role(this, 'mattinstancerole',
      {
        assumedBy: new ServicePrincipal('ec2.amazonaws.com'),
        managedPolicies: [ManagedPolicy.fromAwsManagedPolicyName('AmazonEC2ReadOnlyAccess')],
      });

Matt will spawn in the battleground vpc:

const matt = new Instance(this, "matt", {
      machineImage: linuxImageId,
      instanceType:  iType,
      role: instanceRole,
      vpc: vpcStack.vpc,
      securityGroup: mattSG,
      vpcSubnets: {subnetType: SubnetType.PUBLIC },
    })

Now we have an EC2 instance with a role, which allows to list other EC2 instances. In production you should narrow the resources down to specific instances.

With EC2 Instance Connect i just type:

mssh i-007e6bd0aecd25f71

where i-007e6bd0aecd25f71 is the instance ID of. In the terminal i test whether Matt is really able to list instances:

[ec2-user@ip-10-0-49-237 ~]$ aws ec2 describe-instances --region eu-central-1 | head -n2
{
    "Reservations": [

It works! But is Matt able to keep his secrets for himself…? Meet Jessica Jones from the “Defenders”.

Jessica vs. Matt: 1:0

Jessica Jones is a detective, owner of “Alias investigation”. She is used to sneak into everything, so she manages to get onto the Matt instance.

Now she steals the keys! How to do that?

On Matts EC2 instance the keys are stored in the metadata. She gets the keys with:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/matt-mattinstanceroleFB806583-1R0IS3ZTZG4ZB

Where matt-mattinstanceroleFB806583-1R0IS3ZTZG4ZB is the role name.

As an output you get something like:

{
  "Code" : "Success",
  "LastUpdated" : "2019-09-29T14:05:24Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "ASIAZXXUIISUZ5FLLTAE",
  "SecretAccessKey" : "eb36OnQb9P7+No2rn8tbYm4lur4sNjtxAYfFjrKz",
  "Token" : "AgoJb3JpZ2luX2VjEM7//////////wEaDGV1LWNlbnRyYWwtMSJGMEQCIB8ZibWslHCgHrHgXQWsYX+a/hbpvHUAMBGn+LS/siKnAiBdIAz7roTIuRdSHZNDpjIpmYYSmOZ+nEJkFIxlQo6KZiqBBAin//////////8BEAAaDDY2OTQ1MzQwMzMwNSIMlFHDg9lf8oCVtI3TKtUDVWk9swjngH873C1uovh3EBF8/RK2M5vwx60cwZhKp7STku3vci276soQdz8bIxexnZ6a3UtNcrEXePVdJu8u74rY+JEZmNHMK1h3JSoNN5LEXdQZgB/fGETn1q8RPOw7iAx3bFMov790SsCbm5UXTSzy9pyZ0+LgWt0YUUZKYCvAcbXVuR4XYWwq7mQXZjugFJv08NJaFlmSyCJSk9veOO+K2R5Gc2LVE9sc0x9A5D78QbCB7SJEQHyXH8AN4Jpl0tGl852BPPGW0KrD6OYUzDsYTiIyGMaiN1bDI+w2eedx0PInfEvu0tubB3eiPIbB1YmMKUwVuKJ+QgMWix3QSWjFdqFASzAA2klYhO2uQmmqi0UQR7CVme0ex4s4xqCMWqnGYtbo9gy5ULf0elWAoSrw5KQjzB1iqiIeQGXhcuMsoO1fjkpFcIJJTohwc6UYGYtwrEoSAoYe+5CxkL2y/0UN26DxEeandr/aA7MviOZ20mUGJypkR2dhhqsfRtomdr8OqQn/iLh/H6z6oJzPH6+W/U8E8v3h3a+rvvX70yFnWc4nQGppBaBgTzFbeWhTnZmGIdiZLGDI6XJ/t/4fVzyoSFWvCDb7bYXnRtfdysgfzS3UEDCB9MLsBTq1AUcgEBYGTSX5tOCDYSpBAeY2FXSIN6GBRF1wgwVk3ybKiJWnHePOg6W2cZ7MiQVIXJFGMAXf5KGY5JhpvuJxLafBhu45MIPMaJ7xKriDI7pNDLQ6GBp9XUsW8TjAtjkx942mlE53qg7X8tQzTPAExoPDa1KYdP6EwuR1RwmbwJ0hr8ly53oIXQJm7uoltft67RxBXAK9W+z6lVhqb71P0fecF50AS8nbv6hfZlexwrfNhQxTAzM=",
  "Expiration" : "2019-09-29T20:09:18Z"
}

Jessica takes the Access Key, the secret and the token. The expiration says the key is only valid until 20:09. Its 16:16 So she got a few hours to use the key!

She jumps onto her own instance

mssh ec2-user@i-07553190f25334a3d

And tries the EC2 describe instances:

[ec2-user@ip-10-0-51-109 ~]$ aws ec2 describe-instances --region eu-central-1

An error occurred (UnauthorizedOperation) when calling the DescribeInstances operation: You are not authorized to perform this operation.

So at this moment she is not allowed. Now she exports the stolen secrets into the environment:

export AWS_DEFAULT_REGION=eu-central-1
export AWS_SECRET_ACCESS_KEY=eb36OnQb9P7+No2rn8tbYm4lur4sNjtxAYfFjrKz
export AWS_ACCESS_KEY_ID=ASIAZXXUIISUZ5FLLTAE
export AWS_SESSION_TOKEN='AgoJb3JpZ2luX2VjEM7wEaDGV1LWNlbnRyYWwtMSJGMEQCIB8ZibWslHCgHrHgXQWsYX+a/hbpvHUAMBGn+LS/siKnAiBdIAz7roTIuRdSH...

By doing that she as assumed Matts identity!

She proves that by asking the token service:

[ec2-user@ip-10-0-51-109 ~]$ aws sts get-caller-identity
{
    "Account": "1112223334445",
    "UserId": "AROAZXXUIISUTBRLFHQYW:i-05a7bb3a3883106f0",
    "Arn": "arn:aws:sts::1112223334445:assumed-role/matt-mattinstanceroleFB806583-1R0IS3ZTZG4ZB"
}

(The outputs are not really the same from above - that is just because i did this multiple times)

Now the moment of suspense. Is Jessica now allowed to list the instances with the stolen identity?

[ec2-user@ip-10-0-51-109 ~]$ aws ec2 describe-instances --region eu-central-1 | head -n2
{
    "Reservations": [
...

Yes. She now has succesfully stolen the keys. Thats not good! Or as in the Capitol One breach (see Appendix) - This is really bad.

Welcome on the stage, the Luke, Luke (Cage).

Luke with third line of defense

Luke Cage alias Powerman is undestructable. Nobody gets through his security. But he also has a role and also has credentials, which could be exposed. His secret superpower here is the caller based security. Where Bengston used lambdas injecting security after the instance has started, we just refer to the instance and add a policy to the role, which is refered by the instance profile:

  const callerBasedPolicy = new Policy(this, "callerBasedPolicy");
  const callerBasedPolicyStatement = new PolicyStatement({
    actions: ['*'],
    effect: Effect.DENY,
    resources: ['*'],
  });

  callerBasedPolicyStatement.addCondition('NotIpAddress', {
    "aws:SourceIp" : [
      luke.instancePublicIp,
    ]
  });
  callerBasedPolicy.addStatements(callerBasedPolicyStatement);

  callerBasedPolicy.attachToRole(instanceRole);

With this policy a call to any service (actions: ['*']) is denied, if it is not coming from Lukes ip (luke.instancePublicIp).

You may not add the caller based statement directly to the Luke-Profile. When Luke starts, the policy have to be there. When you create the caller bases policy, the ip of Luke must be there. To avoid the circular dependency, we just add a Policy to the role later, after the instance has started.

And this statement says:

If the call does not come from Lukes IP Address - do not allow it.

Because it is a DENY effect, it has more power than any ALLOW.

Jessica vs. Luke 0:1

Ok, Jessica manages to steal Lukes secrets - watch the series if you want to know how. But as she tries to use the secrets, a surprise is waiting for her:

[ec2-user@ip-10-0-51-109 ~]$ aws sts get-caller-identity
{
    "Account": "1112223334445",
    "UserId": "AROAZXXUIISUTBRLFHQYW:i-05a7bb3a3883106f0",
    "Arn": "arn:aws:sts::1112223334445:assumed-role/luke-lukeinstanceroleFB806583-1R0IS3ZTZG4ZB/i-05a7bb3a3883106f0"
}
[ec2-user@ip-10-0-51-109 ~]$ aws ec2 describe-instances --region eu-central-1 | head -n2

An error occurred (UnauthorizedOperation) when calling the DescribeInstances operation: You are not authorized to perform this operation.

She has managed to steal Lukes identity, but because the call comes from her ip, she gets no access.

Summary

The first line of defense always is the security group with least privileges. I showed you an easy way to automate this. On account level you could/should add NACL as well. (Chemist joke: To salt the food )

The second line of defense here is the policy of the IAM identity you use. And as you surely are aware - with EC2 instances or other AWS services you only use assumed roles, never never static access keys with locally stored credentials.

Here i have showed you how to easily add a third line of defense: Attaching the “where do you come from” question to the policy. So an attacker may get hold of the credentials, but may not use them. Usually this is tricky when you do not automate it. But with the CDK it has become much easier!

Hidden 3 1/2 line of defense: By generating unique names for roles you make it harder to guess the role name. This comes for free with the cdk!

Appendix

Sources

The Technical Side of the Capitol One AWS Security Breach

Instance-Metadata

Dynamically locking credentials to the environment.

Picture

Photo by Touann Gatouillat Vergos on Unsplash