Three hurdles to skip before using the secure Instance Metadata Service V2

Do not use new Instance Metadata Service V2 (imdsv2) without proper prevention!

You may think you can use Instance Metadata Service V2 right away, but there are a few caveats: Many old modules do not work with imdsv2 yet. We look at aws cli, the Systems Manager agent and the Instance Connect service. Currently, these services will not work with imdsv2 on an EC2 instance with the latest Amazon Linux 2 image out of the box. Here you can read how to make them work!

The first step is the precautionary investigation: we will go testing whether these three services are working with imdsv2.

The second step is the cure - how to get them working. Let’s look how its done.

Investigation aws-cli

Setup

In our github repository there is a new “ec2-instanceconnect” template in the repository https://github.com/tecracer/cdk-templates. This template will provide you with an vpc and a EC2 instance for sandbox testing.

First clone the repository, change to the cloned directory.

In the cloned repository change to the dir:

cd ec2-instanceconnect

Create an ssh key

Later you need to access the instance. So create yourself an ssh key and set an environment variable “KEYNAME” to the key name.

We create a key named “blog” with the console like this:

Save the key file locally. Then tell the cdk template about the key name:

export KEYNAME="blog"

Now that we have a key, we can create the aws working infrastructur

Create VPC and EC2 instance with CDK

First, install npm dependencies. You need node and npm installed.

npm i

Deploy the working environment. It is just a vpc with a single instance. You need the tool task only for convenience. All commands can be seen in the file Taskfile.yml.

task deploy

Now we can investigate service one: the AWS CLI.

Investigate aws cli

Now we will test whether the AWS CLI on the instance works with the imdsv2. In the first terminal window, we will connect to the instance. In the second windows we will execute AWS CLO commands locally.

We connect to the instance with the ssh key. First, fetch the public ip of the instance:

aws ec2 describe-instances --query "Reservations[*].Instances[*].[InstanceId,PublicIpAddress]"

Or if you like to use cli v2 with yaml :)

aws2 ec2 describe-instances --query "Reservations[*].Instances[*].[InstanceId,PublicIpAddress]" --output yaml

Put the public ip of the EC2 instance in the local shell environment “IP”, e.g.

export IP=18.196.163.244

Change key permissions

chmod 400 blog.pem

Connect:

ssh ec2-user@$IP -i ./blog.pem

Now we are connected and can test the AWS CLI.

Try on the ec2 instance:

aws sts get-caller-identity

And you will get an answer.

Now, in a second window type (locally) to enable imdsv2 on the instance:

task enable-imdsv2

Back in first window (the instance) make an aws call on the instance again:

aws sts get-caller-identity

You will get:

Unable to locate credentials. You can configure credentials by running "aws configure".

So this version of the AWS CLI is not working with imdsv2.

Cure for AWS CLI

The preinstalled version of AWS cli is currently (on instance):

aws --version

Output: aws-cli/1.16.102 Python/2.7.16 Linux/4.14.154-128.181.amzn2.x86_64 botocore/1.12.92

Please note that this is currently the standard with the newest Amazon Linux 2 AMI!

So you have to install the newest version according to the documentation

We will use the “bundled installer” method. Just execute these commands on the EC2 instance:

curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
unzip awscli-bundle.zip
sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

The new versions works with v2:

/usr/local/bin/aws --version

Gives: aws-cli/1.16.314 Python/2.7.16 Linux/4.14.154-128.181.amzn2.x86_64 botocore/1.13.50

Wenn you call the sts service again, it is working:

/usr/local/bin/aws sts get-caller-identity

You get something like this:

{
    "Account": "666666666666",
    "UserId": "ATXXXYYYZZZAAAUA6W67:i-022eedfddca511df05",
    "Arn": "arn:aws:sts::666666666666:assumed-role/SingleInstanceStack-singleinstanceInstanceRoleFF8A-PDJZROI96CJJ/i-022eedfddca511df05"
}

Now let us see what the Systems Manager (SSM) is saying.

Investigate SSM agent

The instance has a role with the attached policy “AmazonEC2RoleforSSM”. So it should be visible to the SSM.

Lets go to the AWS Systems Manager and click on “Managed Instances”.

We see … nothing.

Systems Manager agent

As for the ssm changelog, ssm Version must be at least 2.3.714.0. The entry says: “Update service domain information fetch logic from EC2 Metadata”

So be carefull to upgrade ssm agent before switching to immdsv2, otherwise systems manager will not see your instance!

Which agent is installed (on instance)?

sudo yum info amazon-ssm-agent

We see that the current version is: Version : 2.3.714.0

Cure for SSM

If we switch of imdsv2 with (locally):

task disable-imdsv2

(Which executes a aws ec2 modify-instance-metadata-options --instance-id i-0ce225ae1d5e67cc4 --http-endpoint enabled --http-token optional), but fetches the instance id from the cdk/cloudformation automatically.)

And maybe restart the agent on the EC2 Amazon Linux 2 instance:

sudo systemctl restart amazon-ssm-agen

The instance will appeard in the “Managed Instances” view in the console. . Or if we want to stay in the (local) cli:

aws ssm describe-instance-information

Gives

{
    "InstanceInformationList": [
        {
            "InstanceId": "i-0ce225ae1d5e67cc4",
            "PingStatus": "Online",
            "LastPingDateTime": 1578761188.386,
            "AgentVersion": "2.3.714.0",
            "IsLatestVersion": false,
            "PlatformType": "Linux",
            "PlatformName": "Amazon Linux",
            "PlatformVersion": "2",
            "ResourceType": "EC2Instance",
            "IPAddress": "10.0.36.102",
            "ComputerName": "ip-10-0-36-102.eu-central-1.compute.internal"
        }
    ]
}

We now issue the run command “AWS-UpdateSSMAgent” on this instance:

aws ssm send-command --document-name "AWS-UpdateSSMAgent" --document-version "1" --targets '[{"Key":"InstanceIds","Values":["i-0ce225ae1d5e67cc4"]}]' --parameters '{"version":[""],"allowDowngrade":["false"]}' --timeout-seconds 600 --max-concurrency "50" --max-errors "0" --region eu-central-1

And after a few seconds the agent has been updated:

 aws ssm describe-instance-information --query "InstanceInformationList[].[InstanceId,AgentVersion]"

Gives

[
   [
       "i-0ce225ae1d5e67cc4",
       "2.3.786.0"
   ]
]

Retest SSM with Instance Metadata Service V2

On the local machine:

task enable-imdsv2

And a retest with

 aws ssm describe-instance-information --query "InstanceInformationList[].[InstanceId,AgentVersion]"

Fine, cured.

That was easy, noe for the hardest part, the Instance Connect service.

Investigate Instance Connect

If you want to use the latest development in EC2 security, the instance metdataservice V2 you will notice a small sentence in the AWS instance connect documentation: if you configure the instance metadata service to require Instance Metadata Service Version 2, you can’t use EC2 Instance Connect.

We switch imdsv2 off (locally):

task disable-imdsv2

And check that instance connect is working (locally):

task connect

(Which executes just mssh i-0ce225ae1d5e67cc4). You have to have AWS EC2 Instance Connect CLI installed. See on github. Or just do pip install ec2instanceconnectcli if python/pip is installed locally.

When we switch imdsv2 on again: task enable-imsdv2, the task connect gets a Permission denied error.

Cure for Instance Connect

I habe patched the aws-ec2-instance-connect-config. use at your own risk, this is no offical AWS patch.

Connect to the ec2 instance with ssh:

ssh ec2-user@$IP -i ./blog.pem

Install git:

sudo yum install git -y

Clone the repo:

git clone https://github.com/tecracer/aws-ec2-instance-connect-config.git

Change into repo :

cd aws-ec2-instance-connect-config

Building patched package for instance-connect

Install additional development tools:

sudo yum install -y rpmdevtools

Build rpm package

make rpm

Now you should have the file ec2-instance-connect-1.1-13.noarch.rpm

This is the package we install now:

sudo yum install ./ec2-instance-connect-1.1-13.noarch.rpm
sudo systemctl daemon-reload

Just to be sure you should always run the instance with an ssh key at first if the instance connect does not work.

Now switching back to local, the connect works:

task connect
mssh i-0ce225ae1d5e67cc4
Last login: Sat Jan 11 16:58:46 2020 from i577bc829.versanet.de

Or the whole picture:

Explanation for instance connect package

What does the instance connect package do?

On the instance in the sshd configuration file (sudo less /etc/ssh/sshd_config), the last lines start the helper script, wich should get the ssh key from the AWS hypervisor:

AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys %u %f
AuthorizedKeysCommandUser ec2-instance-connect

The old version (Github eic_curl_authorized_keys) tries to get the key in the immdsv1 way:

curl_cmd="/usr/bin/curl -s -f -m 1"

The V2 way

This is done at several lines in the scripts. The patch changes that in several places to:

TOKEN=`/usr/bin/curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
curl_cmd='/usr/bin/curl  -H "X-aws-ec2-metadata-token: $TOKEN" -s -f -m 1 '

Which is the imdsv2 way.

Update Jan/20/2020: The official Version now also has support for EC2 Instance Metadata Service version 2. This is Version ec2-instance-connect 1.1.12. But it may not available as rpm already.

Summary

I have shown you, that AWS CLI, Systems Manager and Instance Connect all do not work out of the box. But as you have seen, there are cures!

Now we can work with the more secure instance metadata Version 2. Please take care of own software or user software which might be using older versions of the AWS sdks. Test this before using V2.

Have fun testing!

Appendix

1. Repository with infrastructure scripts:

https://github.com/tecracer/cdk-templates

2. Repository with patched instance connect

https://github.com/tecracer/aws-ec2-instance-connect-config

3. Instance Connect cli

https://github.com/aws/aws-ec2-instance-connect-cli

Photos

Photo by Interactive Sports on Unsplash