Views of the Pyramids: From a monolithic Test process to a Serverless Test Automation with CodeBuild
Comparing the development methodology of a monolithic program to a Serverless IAC application you will see that the power of DevOps lies in automating everything. I will show you a working example of a serverless CI pipeline with automated unit, integration and end2end test and test reports in CodeBuild. The full source is written GO, with references to Node.JS and python for the test parts.
This post focuses on the testing patterns and the implementation with CodeBuild. The CDK and Lambda code is explained in detail on go-on-aws Serverless Testing Pyramid.
Monolithic development
I am descriping a typical development methodology of a project that I used to know (tm):
Development
Several developers (Dev) coded several parts of the application. Integration of the parts is done by a local Jenkis Server. The developers perform manual checks of their application against a test database. The do not have access to the creation and managing the configuration of the infrastructure.
The have a code repository where they store the code. For a release, they create an artifact like compiled applications, installation scripts and SQL code to change the database to the new release. This artifact is given to the Test or QA department.
Testing
The installation scripts are executed in the test server and test database.
For a release, there is a test plan with defined test cases, which are manually performed. Results of failed test cases are reported back to the developers until all test passes.
If infrastructure is changed, like the database configuration and indexing, this is performed before the test. So in a way infrastructure is tested. The big difference to infrastructure as code is, that a change of the server hardware is not part of any development cycle. The size is planned upfront and just has to fit. Upscaling of server hardware is only a task of the operations department.
Testing is done according to ISTQB, see some of the test types in the online ISTQB Glossary.
Operations
After successfull testing the artifacts (often not the same files as used for testing, but specially generated for production) are given from the test department to the operations team.
During a maintanance window (of several hours) the script and changes are applied and some basic end2end test are performed.
This is the “old world”.
Now we have a look at Test automation for a serverless application
Testtypes for Serverless Testing - Theory
Unit Testing
For the application and the infrastructure, there should be automated unit testing. This can be a totally separate test. Application and infrastructure are not integrated yet.
Integration Testing
Ask yourself: Which separated parts/components are now working together? If you cannot test the components individually, they are to tightly coupled together.
Tightly-coupled component depend on each other and are not usable independently. So the testability is a hint to good system design.
If components are loosely coupled, you can test the interface.
For a serverless application on AWS usually you have AWS Resources like storage (like S3) or databases (like dynamodb) wich are a part of the application. But these parts should be loosely coupled, so that you could exchange components and test single components. For each component of the application you have to decide whether it should be interchangeable or not.
The integration test can be performed only on the application side, when different component of the “software” (application) part are integrated. Also the “software” is integrated with the “hardware” (infrastructure).
End2End Testing
End2End means performing a test from the start event is as realistic as possible - this is the first end: End2End. Then one path of the application is tested with variations of parameter and the result is checked against the expected results. This is the other end: End2End.
Also this is a bet on how the production environment will behave. So you will only come very near to realistic production environment, 100% is almost impossible to reach.
Testtypes for Serverless Testing - Serverless DSL application
I show you the automation on a standard DSL - Dynamo S3 Lambda application. In 2018 my fellow consultant Marco Tesch had the idea to define benchmarks for IaC scenarios. We take the serverless application scenario. See the Code on Github
The Use Case description:
- User uploads object to S3 bucket
- Bucket upload event goes to lambda
- Lambda writes object name with timestamp to dynamoDB
As we have a Retention Policy, CDK creates a helper Lambda as well.
Unit Testing
Infrastructure Unit Tests
Unit testing for the CDK can be the test of the generated CloudFormation. This has two main purposes:
- Test if the iac code builds correctly
- Test if the right CloudFormation is generated
Test if the iac code builds correctly
CDK is very volatile, you have new versions each week, interfaces are changing from unstable to stable and so on. In the early days of the CDK around v. 0.35 i checked this in: CDK - under Construction - should we use it for the next project?
Test if the right CloudFormation is generated
We are using program logic to dynamically generate CloudFormation. So we should write test.
See go-on-aws for a deep dive on the code.
Application Unit Test
If the application is not to tighly coupled, this test type should be easy to implement. Basic business logic should not depend on AWS Services. E.g. if you use Systems Manager Parameter store or App Config for configuration, then there should be an abstraction to this configuration in your business logic.
So in the picture the grey Infrastructure part can be viewed seperated from the Application part. Here we have only a small part of logic as an example.
See go-on-aws for a deep dive on the code.
Integration Testing
Infrastructure Integration Test
Here I test whether the CDK code really created AWS Resources. You could discuss whether this really can be called “integration”, but it moves the test to higher levels of the test pyramid, because its closer to production.
The purpose here is not to check, whether an “AWS::Lambda::Function” Cloudformation really creates a function. This is a safety net whether I selected the correct Constructs and you also get insights about timing of the creation.
See go-on-aws for a deep dive on the code.
Application Integration Test
In the first part I only want to test the functionality of the Lambda function running really on AWS. This also test the IAM rights of the function, some timing and configuration etc.
So I feed a test event directly into Lamba. This way the test does not depend on the right configuration of the S3 events, you just test the lambda itself.
For the output I weigh effort against benefit. Beeing a purist you should mock the DynamoDB table for lambda. Pragmatically I just check the Table itself.
See go-on-aws for a deep dive on the code.
End2End Testing
Now the whole chain is tested. The test puts a file directly in the bucket and checks the Table:
- Setup (delete testitem in table)
- Trigger end2End event (put file in S3 Bucket)
- Test (check table entry)
- Teardown / Cleanup (delete testitem in table)
See go-on-aws for a deep dive on the code.
Automate Everything
With AWS CodeBuild, you can setup a serverless CI build environment. All test can be performed by CodeBuild and you see the test results aka reports in the AWS console also. See AWS documentation create a build project for details. For the start it is easier to create the CodeBuild build project with the AWS console, because you have guiding wizards and the intial roles are created automatically.
The various test output formats from the several test frameworks from Python, Node.JS and GO are not natively supported. But as the Java testing library “Junit” is the de-facto standard, you can convert all types to Junit.
See AWS Documentation Working with test reporting in AWS CodeBuild for more detail.
Python
Pytest supports junitxml:
python -m pytest --junitxml=<test report directory>/<report filename>
See AWS documentation Set up test reporting with pytest
GO
You can use go-junit-report to convert the GO standard test outputs to JUNITXML:
As an example:
go get -u github.com/jstemmer/go-junit-report
I_TEST=yes go test -v 2>&1 | go-junit-report >$CODEBUILD_SRC_DIR/report-infra-integration.xml
Here I control the test type with the I_TEST environment variable.
Node.JS
You can use Jest with the jest-junitreporter.
See AWS documenation Set up test reporting with Jest
for details.
CodeBuild
I use the managed Ubuntu image for the Build Project:
The automation steps are defined in the buildspec file. This file tells CodeBuild what steps are performed. Here i have the full example for the example with CDK and Lambda function in GO:
Buildspec
I am not deploying anything, in the buildspec only the tests are defined.
Install Phase
version: 0.2
2
3 phases:
4 install:
5 runtime-versions:
6 golang: 1.16
7 nodejs: 14
8 commands:
9 - echo Installing CDK..
10 - npm i cdk@v2.1.0 -g
11 - go get -u github.com/jstemmer/go-junit-report
We need GO and nodejs. Here you could also define the CDK version as a variable to test new CDK versions.
Build Phase
Unit Test for the application
12 build:
13 commands:
14 - echo Unit Testing app...
15 - cd $CODEBUILD_SRC_DIR/architectures/serverless/app
16 - go test -v 2>&1 | go-junit-report >$CODEBUILD_SRC_DIR/report-app.xml
Unit Test for the Infrastructure
17 - env GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o dist/main main/main.go
18 - chmod +x dist/main
19 - cd dist && zip main.zip main
Prepare the Lambda deployment package, because the CDK looks for this.
20 - echo Unit Testing infra...
21 - cd $CODEBUILD_SRC_DIR/architectures/serverless/infra
22 - go test -v 2>&1 | go-junit-report >$CODEBUILD_SRC_DIR/report-infra.xml
Perform the CloudFormation unit tests.
See code on github
Integration Test for CDK / Infrastructure
23 post_build:
24 commands:
25 - echo Deploying infra
26 - cd $CODEBUILD_SRC_DIR/architectures/serverless/infra
27 - cdk deploy --require-approval never
28 - echo Integration Testing infra...
29 - export I_TEST=yes
30 - env I_TEST=yes go test -v 2>&1 | go-junit-report >$CODEBUILD_SRC_DIR/report-infra-integration.xml
31 - cd $CODEBUILD_SRC_DIR/architectures/serverless/app
32 - echo Integration Testing App...
33 - env I_TEST=yes go test -v 2>&1 | go-junit-report >$CODEBUILD_SRC_DIR/report-app-integration.xml
34 - cd $CODEBUILD_SRC_DIR/architectures/serverless/infra
35 - echo Destroying infra
36 - cdk destroy -f
Here the whole infrastructure is built, tested and destroyed again. You have to add some CDK cli switches so that CodeBuild does not wait for you to press “Y” in lines 27 and 36.
If you want to do the same for terraform, use terratest.
Define Test Reports
The units test have written test reports. Now you tell CodeBuild where to find them:
39 reports:
40 gotest_reports:
41 files:
42 - report-app.xml
43 - report-infra.xml
44 - report-app-integration.xml
45 - report-infra-integration.xml
46 base-directory: $CODEBUILD_SRC_DIR
47 file-format: JUNITXML
Build Policies
Please note the difference for the IAM rights. Because the CDK creates the infrastructure, the role for CodeBuild must have the matching IAM policies for that.
Testresults
In the build history you see the results of the build itself. In this example we have a failed test in the beginning.
The overall status of the test shows “failed”:
Looking at the test details we see that the infrastructre integration test “TestInfraLambdaExist” failed. This test checks whether the CDK really created a lambda function or just pretended to do so
So we only have 90% pass rate:
Resolving an Error
The details of the failed test shows, that the test itself does not have enough IAM access rights:
integration_test.go:18: assertion failed: error is not nil: operation error Lambda:
GetFunction, https response error StatusCode: 403,
RequestID: 2fadad1e-f943-41f0-b147-2e9a805c36da,
api error AccessDeniedException: User: arn:aws:sts::555555555555:assumed-role/codebuild-go-on-aws-source-service-role/AWSCodeBuild-aee11252-e13c-4e4a-bda1-15f3864a87f2
is not authorized to perform: lambda:GetFunction on resource:
arn:aws:lambda:eu-central-1:555555555555:function:logincomingobject
because no identity-based policy allows the lambda:GetFunction action: GetFunctionConfiguration should return no error
So I add the policy to the codebuild-go-on-aws-source-service-role Role and run the build/test process again:
Summary & Outlook
I have shown the possibility for a small project to get a good test coverage not only on the application part to inspire you to think about different views for automated test of the applications as a whole.
In real projects you would not directly deploy this tested application to production. You would also add QS steps e.g. with the customer. But as you have automated all test, the speed and agility is much faster as in the monolithic setup.
To get ideas about how to shape the development phases I recommend having a look at Emily Freemans new model for software development.
Thanks for reading, I hope you got some new ideas and insights for better application quality!
See the full source on github.
Feedback & discussion
For discussion please contact me on twitter @megaproaktiv
Learn more GO
Want to know more about using GOLANG on AWS? - Learn GO on AWS: here
