Getting around circular CloudFormation Dependencies: S3-Event-Lambda-Role
This content is more than 4 years old and the cloud moves fast so some information may be slightly out of date.
Getting around circular CloudFormation dependencies
Several posts complain about the inability of CloudFormation to apply a Lambda event function to an S3 Bucket with an dynamically generated name.
The standard UseCase is an S3 Bucket with a Lambda event notification. In this special case the Bucket has a dynamically generated name. This cannot be done by pure CloudFormation!
How to work around this circular depency? Let me show you an easy way:
In AWS Blog: Handling circular dependency errors in AWS CloudFormation [AWS1] and serverless hero Ben Kehoe writes here .[BEN1]
This will not work
So in pure CloudFormation this is not possible:
That is because we have a circular dependency. As [AWS1] says: “The bucket notification is dependent on the Lambda function and the Lambda function is dependent on the execution role, which is dependent on the S3 bucket.”
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
Bucket:
Type: AWS::S3::Bucket
Function:
Type: AWS::Serverless::Function
Properties:
CodeUri: s3://bucketname/object.zip # Add in an S3 URI where you have code Lambda Code
Runtime: python2.7
Handler: index.handler
Policies:
- Version: 2012-10-17
Statement:
- Effect: Allow
Action: s3:GetObject*
Resource: !Sub "arn:aws:s3:::${Bucket}*"
Events:
Upload:
Properties:
Bucket:
Ref: Bucket
Events: s3:ObjectCreated:*
Type: S3
So we have to decouple things. We can do that by first creating the bucket and the lambda function and bringing them together after that.
The way to extend CloudFormation is a custom resource. That is a Lambda Function wich acts as a CloudFormation resource. This “bring together” function gets notified to create or update or delete itself.
You could also do this as an AWS cli script after creation of the CloudFormation stack. But if you do that, you won’t have the stack itself as a whole functional unit. But beeing lazy, the question is: do I really have to code this all by myself? - Answer is no - CDK to the rescue.
This will work
CDK code
If you create an CDK app with cdk init app --language=typescript you just have to get some imports:
import {Bucket,EventType} from '@aws-cdk/aws-s3';
import {LambdaDestination} from '@aws-cdk/aws-s3-notifications';
import {Function, Runtime, Code} from '@aws-cdk/aws-lambda'
And replace // The code that defines your stack goes here with this snippet:
export class CfnGraphStack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
// S3
const bucket = new Bucket(this, "testbucketmpa",{
removalPolicy: cdk.RemovalPolicy.DESTROY, // NOT recommended for production code
});
// Lambda
const lambda = new Function(this, 'HelloHandler', {
code: Code.asset(path.join(__dirname, '../lambda')),
handler: 'hello.handler',
runtime: Runtime.NODEJS_8_10,
memorySize: 1024
});
// S3 -> Lambda
bucket.addEventNotification(EventType.OBJECT_CREATED, new LambdaDestination(lambda));
}
}
Of course there have to be a node Lambda function in the dir lambda…
The CDK has already added the “bring together” Lambda function. And it is all happening under the hood, we just add the EventNotifcation in the ts code to the Bucket.
Generated template
testbucketmpaNotificationsB2722499:
Type: Custom::S3BucketNotifications
Properties:
...
BucketName:
Ref: testbucketmpaAE8E5392
NotificationConfiguration:
LambdaFunctionConfigurations:
- Events:
- s3:ObjectCreated:*
LambdaFunctionArn:
Fn::GetAtt:
- HelloHandler2E4FBA4D
- Arn
DependsOn:
- HelloHandlerAllowBucketNotificationsFromCfnGraphStacktestbucketmpa0C1D72A46851358B
This snippet from the generated CloudFormation (cdk synth) shows the Custom Ressource which waits (with DependsOn until the AWS::Lambda::Permission is created.)
The Lambda Function is created inline:
BucketNotificationsHandler050a0587b7544547bf325f094a3db8347ECC3691:
Type: AWS::Lambda::Function
Properties:
Description: AWS CloudFormation handler for "Custom::S3BucketNotifications" resources (@aws-cdk/aws-s3)
Code:
ZipFile: >-
exports.handler = (event, context) => {
const s3 = new (require('aws-sdk').S3)();
const https = require("https");
const url = require("url");
log(JSON.stringify(event, undefined, 2));
...
return s3.putBucketNotificationConfiguration(req, (err, data) => {
Its creates the BucketNotificatioConfiguration and you’re done!
