CDK Infrastructure Testing - Part 2a - Implement Unit, Integration and Application Test for CDK Infrastructure and an EC2 Web Server Application

With CDK you create Infrastructure as Code - IaC. You can automate the test for the IaC code. The three test tastes -Unit, Integration and Application- should work closely together. Here I show you how. It is like the three steps of coffee tasting: 1 smell, 2 Taste, 3 Feel.

You can start immediately with the GO implementation or achieve the same effect in another CDK-supported language such as TypeScript or Python by following the steps described. The cit GO integration and application tests are directly applicable to all CDK templates, no matter which programming language is used.

With a few lines of code, you got tests for the stack level, the physical resource level and the application. Let`s apply that to a CDK generated EC2 Load Balancer App.

From Unit test to application test with a Load Balancer EC2 App

We will go through three test types in this post

1. The Unit test
2. The physical resource test called cit - CDK Infrastructure Testing
3. Application Test

Overview

Unit Test: The generated CloudFormation (Cfn) is tested. Usually, you can rely on the fact that e.g. the SNS Construct generates an SNS Resource, but… When you use some programming logic inside the Construct, you can not be 100% sure that the right AWS resources are generated. This is what Unit testing is made for.

Integration Test: Resource creation is tested. Sometimes on Monday, I ask myself the question “did I really deploy that last Friday?” - this happens when you run out of coffee. So the next level is to test whether the resource in the Construct really is created. To get traceability I want to use the given Construct ID to access the physical resource. Traceability means that you know which Resource is created by which Construct.

Application Test: The functionality of the application is tested. With an AWS application bundled with infrastructure, a new version of your app does not only consist of the software itself but also the changes in the infrastructure. So it makes sense to test the responses from the application to certain requests.

Let’s walk through the steps. We use go/alb_ec2 from the repository https://github.com/tecracer/cdk-templates. You will find the same CDK template in typescript/alb_ec2. Still, somebody has to code the python example…

CDK generated Web Server.

1 Smell: Unit Test - Template creation

As discussed in Part 1, the standard Unit Tests checks the CDK generated CloudFormation template.

We create an Application Load Balancer with the ConstructID LB

This name should be meaningful to you. I like names short and sweet, so “LB”. This id will be used for all test types. You do not need to create Systems Manager Parameters or CloudFormation Exports like discussed in part 1, just use the Construct ID.

This is the Load Balancing Construct in GO CDK:

lb := elasticloadbalancingv2.NewApplicationLoadBalancer(stack, aws.String("LB"),
  &elasticloadbalancingv2.ApplicationLoadBalancerProps{
    Vpc:              myVpc,
    InternetFacing:   aws.Bool(true),
    LoadBalancerName: aws.String("ALBGODEMO"),
  },
)

While it is a string 'LB' in TypeScript, GO uses String pointers for efficiency. aws.String("LB") creates a string pointer.

This is the Load Balancing Construct in TypeScript CDK:

const lb = new ApplicationLoadBalancer(this, 'LB', {
    vpc: albVpc,
    internetFacing: true
    
  });

This is the Load Balancing Construct in Python CDK:

lb = elbv2.ApplicationLoadBalancer(self, "LB",
        vpc=vpc,
        internet_facing=True
    )

The second parameter is the Construct ID.

With the Construct defined, we can call cdk synth. This synthesizes the CloudFormation template in the directory cdk.out.

Tipp: Use npx cdk@v2.0.0-rc.8 instead of cdk which will call the TypeScript transpiler , so you do no need npm build before.

In the CloudFormation template cdk.out/AlbInstStack.template.json this is the generated code for the Load Balancer:

"LB8A12904C":{
  "Type": "AWS::ElasticLoadBalancingV2::LoadBalancer",
  "Properties": {
    "LoadBalancerAttributes": [
      {
        "Key": "deletion_protection.enabled",
        "Value": "false"
      }
    ],
    "Name": "ALBGODEMO",

Where LB8A12904C is the Logical ID.

We define the Unit Test for that in GO:

func TestAlbInstStack(t *testing.T) {
	// GIVEN
	app := awscdk.NewApp(nil)

	// WHEN
	stack := alb_ec2.NewAlbEC2Stack(app, "MyStack", nil)

	// THEN
	bytes, err := json.Marshal(app.Synth(nil).GetStackArtifact(stack.ArtifactId()).Template())
	if err != nil {
			t.Error(err)
	}
  // Check
	template := gjson.ParseBytes(bytes)
	albName := template.Get("Resources.LB8A12904C.Properties.Name").String()
	assert.Equal(t, "ALBGODEMO", albName)
}

1. Given: An CDK app is created
2. When: The stack is created
3. Then: The data structure “StackArtifact” is translated to json. This is called “marshaling”
4. Check: the gson library is used to parse the json file

This is the part which you also can do in TS/Python. The cdk init app generates a test skeleton for you.

How do you know the name “LB8A12904C”? - Answer is you don’t. You have to synthesize once and take the Logical ID out of the template file.

We run the test:

go test -run TestAlbInstStack -v
=== RUN   TestAlbInstStack
--- PASS: TestAlbInstStack (7.82s)
PASS
ok  	alb_ec2	8.430s

This test can not only be used for testing created parameters e.g. for the albName. It also proofs that the build process runs without errors. Try it out and change the name of the Load Balancer or change the Construct ID. The test will FAIL.

In TypeScript this test would look like:

test('Load Balancer exists', () => {
    const app = new cdk.App();
    // WHEN
    const stack = new AlbEc2.AlbEc2Stack(app, 'MyTestStack');
    // THEN
    const actual = app.synth().getStackArtifact(stack.artifactId).template;
    expect(actual.Resources.LB8A12904C.Type).toEqual("AWS::ElasticLoadBalancingV2::LoadBalancer")
});

To be exact this TypeScript test only checks the Type, not the property.

Currently, the integration and applications tests are failing:

go test  -v
=== RUN   TestALBRequest
    ssm.go:18:
        	Error Trace:	ssm.go:18
        	            				alb_ec2_test.go:35
        	Error:      	Received unexpected error:
        	            	ParameterNotFound:
        	            		status code: 400, request id: a9b1f50b-c462-4b9b-9bf4-72fbb4768114
        	Test:       	TestALBRequest
--- FAIL: TestALBRequest (0.37s)
=== RUN   TestAlbPhysicalResource
FATA[0000] Template AlbInstStack not found
exit status 1
FAIL	alb_ec2	1.145s

This is correct because we do not have a physical resource for the Load Balancer yet.

We have tested the CloudFormation Template with a Unit test.

With a cdk deploy the template is sent to the CloudFormation service. CloudFormation generates the Stack, which includes the physical resource “load balancer”. I think it’s funny to talk about “physical” resources in the Cloud, but that is the AWS wording :).

2 Taste: Integration test

Now with cdk deploy the resources are generated. CDK adds all necessary auxiliary resources, which the Load Balancer needs to work.
So we do not have to define everything in the Construct. All the physical resources together build the CloudFormation stack.

Different resources from the stack can have different states during creation:

cdkstat AlbInstStack
Logical ID                       Pysical ID                       Type                             Status
----------                       ----------                       -----------                      -----------
ASG46ED3070                      autoscalingGroupCDKDEMO          AWS::AutoScaling::AutoScalingGr  CREATE_IN_PROGRESS
ASGInstanceProfile0A2834D7       AlbInstStack-ASGInstanceProfile  AWS::IAM::InstanceProfile        CREATE_COMPLETE
ASGInstanceSecurityGroup0525485  sg-0bb8c50c146e319d7             AWS::EC2::SecurityGroup          CREATE_COMPLETE
ASGInstanceSecurityGroupfromAlb  ASGInstanceSecurityGroupfromAlb  AWS::EC2::SecurityGroupIngress   CREATE_COMPLETE
ASGLaunchConfigC00AF12B          AlbInstStack-ASGLaunchConfigC00  AWS::AutoScaling::LaunchConfigu  CREATE_COMPLETE
CDKMetadata                      d650d230-cfa0-11eb-b478-06e6f2d  AWS::CDK::Metadata               CREATE_IN_PROGRESS
...

Each resource in the CloudFormation stack starts with the state CREATE_IN_PROGRESSand hopefully ends with CREATE_COMPLETE

When all resources have the state CREATE_COMPLETE, the stack is completed. The Load Balancer should be created. To be sure, we test that.

The test for the physical resource looks like:

func TestAlbPhysicalResource( t *testing.T){
	if testing.Short() {
        t.Skip("skipping integration test in short mode.")
    }
	alb, err := citalbv2.GetLoadBalancer(aws.String("AlbInstStack"), aws.String("LB"))
	assert.NilError(t,err,"The LoadBalancer should be retrievable without error")
	// Just read anything from alb
	applicationType := awselbv2types.LoadBalancerTypeEnumApplication
	assert.Equal(t, applicationType, alb.Type)
}

The GetLoadBalancer takes care of the translation from “Construct with ID LB from Stack AlbInstStack” to an Load Balancer data structure .

With the AWS cli call:

aws cloudformation describe-stack-resource  --stack-name AlbInstStack --logical-resource-id LB8A12904C

you can see the CloudFormation data of the physical resource:

  "StackResourceDetail": {
        "StackName": "AlbInstStack",
        "StackId": "arn:aws:cloudformation:eu-central-1:555544446666:stack/AlbInstStack/d650d230-cfa0-11eb-b478-06e6f2d05224",
        "LogicalResourceId": "LB8A12904C",
        "PhysicalResourceId": "arn:aws:elasticloadbalancing:eu-central-1:555544446666:loadbalancer/app/ALBGODEMO/fa33a9bde8742fe6",
        "ResourceType": "AWS::ElasticLoadBalancingV2::LoadBalancer",
        "LastUpdatedTimestamp": "2021-06-17T19:22:01.310000+00:00",
        "ResourceStatus": "CREATE_COMPLETE",
        "Metadata": "{\"aws:cdk:path\":\"AlbInstStack/LB/Resource\"}",
        "DriftInformation": {
            "StackResourceDriftStatus": "NOT_CHECKED"
        }
    }

The citalbv2.GetLoadBalancer uses this CloudFormation data to get the Load Balancer Data with the GO SDK.

Let us have a look at the physical test:

1. if testing.Short()

If you call go test -short the short flag will be set and the test will be skipped. This is useful if you not have deployed the stack yet, so you know the test will fail.

2. The assert.NilError checks wether the resource is really there.

3. Check fields, applicationType := awselbv2types.LoadBalancerTypeEnumApplication As an example the type is checked whether it is really is an Application Load Balancer.

So you can check for the data fields of the resource itself - not the CloudFormation data.

Some of the data fields are:

AvailabilityZones []AvailabilityZone
CanonicalHostedZoneId *string
CreatedTime *time.Time
IpAddressType IpAddressType 
LoadBalancerArn *string
LoadBalancerName *string
SecurityGroups []string
State *LoadBalancerState
Type LoadBalancerTypeEnum

If you want to test some connected resources, you retrieve them with the SDK. If you want to check how many Security Groups are attached, you do that directly on SecurityGroups []string with len(alb.SecurityGroups).

The shortest integration test - check for existence - is just:

alb, err := citalbv2.GetLoadBalancer(aws.String("AlbInstStack"), aws.String("LB"))
assert.NilError(t,err,"The LoadBalancer should be retrievable without error")

To separate the test levels, you could also use tags in GO. You use different files for the tests and tag them like:

// +build integration

package alb_ec2

to include.

With go test --tags=integration you would only run test files tagged with integration

This test written in GO can also be applied to CDK generated CloudFormation stacks from other programming languages! At first sight, the idea to write a test in a different language seems strange. But this is the same as what you would do if using Chef inspec which uses Ruby. The difference is that you do not use a DSL (Domain Specific Language), but directly work on the AWS GO SDK. With a DSL you have limited possibilities, which the SDK you can test anything.

We have tested the physical Load Balancer resource with an integration test.

3 Feel: Application Test

func TestALBRequest(t *testing.T) {
	if testing.Short() {
        t.Skip("skipping integration test in short mode.")
    }
	storedUrl := terratest_aws.GetParameter(t,region,"/cdk-templates/go/alb_ec2")

	url := fmt.Sprintf("http://%s", storedUrl)

	sleepBetweenRetries, error := time.ParseDuration("10s")

	if error != nil {
		panic("Can't parse duration")
	}

	http_helper.HttpGetWithRetry(t, url, nil, 200 , "<h1>hello world</h1>", 20, sleepBetweenRetries)
	
}

This is almost the same as in part1 blogpost. We are using terratest to send http requests. If we want the test for certain data in the response, you can add a helper function. See https://github.com/gruntwork-io/terratest/tree/master/modules/http-helper for more details.

We have tested the application.

The CIT lib

This is a GO implementation of the concept to take the CDK Construct ID as the identifier for all test levels. If you want to code it in your language, this should be doable with these hints.

You can use the GO module from https://github.com/megaproaktiv/cit

How do you get the physical ID from the Construct id?

Let us have a look at the ALb CloudFormation:

"LB8A12904C": {
    "Type": "AWS::ElasticLoadBalancingV2::LoadBalancer",
    "Properties": {
      "LoadBalancerAttributes": [
        {
          "Key": "deletion_protection.enabled",
          "Value": "false"
        }
      ],
...
    "Metadata": {
      "aws:cdk:path": "AlbStack/LB/Resource"
    }
  },

To know which Construct belongs to which resource, the CDK has to maintain the mapping state. It is stored in the Metadata:

"aws:cdk:path": "AlbStack/LB/Resource"

The first part is the stack-name, the second is the Construct id, then “Resource”.

The algorithm:

1. Get the template from CloudFormation with the template name and the GetTemplate API call
2. Find the ConstructID name from the metadata. AlbStack/LB/Resource
3. Get the Logical ID from the Resource LB8A12904C
4. Call the DescribeStackResource API call with the Logical ID - this will give you the Physical ID

See the GO code: PhysicalIDfromCID

1. Get template

	resGetTemplate, err := client.GetTemplate(context.TODO(), parameterStack)

2. / 3. Find Cid / Get the Logical ID

for key, resource := range stack.Resources {
  if resource.Metadata != nil {
    if resource.Metadata["aws:cdk:path"] != "" {
      meta := resource.Metadata["aws:cdk:path"]
      log.Debug("Path: ",meta)
      templateConstructID := ExtractConstructID(&meta)
      if templateConstructID == *constructID {
          return &key, nil
      }
    }
  }
}

4. DescribeStackResource

resourceDetail,err := client.DescribeStackResource(context.TODO(), parameterResource)
if err != nil {
  return nil, err
}
// find physicalid
physicalId := resourceDetail.StackResourceDetail.PhysicalResourceId

Low Level helper functions

If you want to check any AWS resource, you can use the PhysicalIDfromCID function, which implements the matching. When you insist on not using go, just implement it for the language and testing framework of your choice.

Higher Abstraction

During the last weeks, i implemented functions like

• GetLoadBalancer
• GetUser (iam)
• GetVpc, GetSecurityGroup and of course for Lambda
• GetFunctionConfiguration

Although it is easy to implement the function for several resources, a simpler call like just GetLoadBalancer is nice. Please contact me for adding other resources, because there are many resources and I will not add all of them in this lifetime.

In the next part we will apply this to Lambda functions.

The End

The integration of all test types together has many advantages in my opinion. What is your opinion on this? Is it helpful for your project? I would be happy to hear your experiences!

I hope this concept or the cit framework implementation will help you with your projects also. For the last couple of projects, I started with an integrated or application test and found it quite useful to get the rights results and staying focused.

Some of the other integration test i used were:

• AWS Workspaces and Workspaces User creation
• AWS Transfer sftp User and read/write test
• Application Load Balancer with Domain and installed software

For discussion please contact me on twitter @megaproaktiv

Appendix

The repositories

Cit - CDK Integration Testing

• https://github.com/megaproaktiv/cit

cdkstat - Show CloudFormation Stack status

• https://github.com/megaproaktiv/cdkstats

CDK Templates using CIT and terratest for testing

• https://github.com/tecracer/cdk-templates

Terratest

Quick Start

• https://github.com/gruntwork-io/terratest

The tools

Awsume

• awsume
• Taskfile

Thanks

Photo by Nathan Dumlao on Unsplash