Deployment Issues with Cross Stack Dependencies and the CDK

Introduction

Today I’m going to share with you a problem I encountered when working with the CDK and cross-stack-references. Just sharing problems wouldn’t be particularly useful though, so I’m going to show you how to solve the problem as well.

Setup

We start with a fairly simple setup with two stacks. Stack 1 is called Exporting Stack and creates two IAM-Roles:

class ExportingStack(core.Stack):

    exported_role_a: iam.Role
    exported_role_b: iam.Role

    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        self.exported_role_a = iam.Role(
            self,
            "exporting-role-a",
            assumed_by=iam.ServicePrincipal("ec2.amazonaws.com")
        )

        self.exported_role_b = iam.Role(
            self,
            "exporting-role-b",
            assumed_by=iam.ServicePrincipal("ec2.amazonaws.com")
        )

The second stack is the importing stack, which creates an S3 bucket with a bucket policy that references these roles.

class ImportingStack(core.Stack):

    def __init__(
        self,
        scope: core.Construct,
        id: str,
        role_a: iam.Role,
        role_b: iam.Role,
        **kwargs
    ) -> None:
        super().__init__(scope, id, **kwargs)

        test_bucket = s3.Bucket(
            self,
            "some-bucket",
            removal_policy=core.RemovalPolicy.DESTROY
        )

        test_bucket.add_to_resource_policy(
            iam.PolicyStatement(
                actions=["s3:GetObject"],
                principals=[
                    role_a,
                    role_b
                ],
                resources=[
                    test_bucket.arn_for_objects("*"),
                    test_bucket.bucket_arn
                ]
            )
        )

Both stacks are initialized like this:

app = core.App()
export = ExportingStack(app, "export")

ImportingStack(
    app,
    "import",
    role_a=export.exported_role_a,
    role_b=export.exported_role_b
)

The initial deployment of these stacks works exactly as you’d expect. CDK recognizes, that there is a dependency between the two stacks and creates the exporting stack first. The exporting stack exports the ARNs of the two roles it creates. Afterwards the CDK deploys the importing stack, which makes use of the exports from the first stack using the Fn::ImportValue intrinsic function in CloudFormation to create the bucket with the bucket policy that references those two ARNs. This creates a hard dependency between the two stacks - the exporting stack may not change the value of the export, i.e. the ARN as long as other stacks import that value.

The architecture we have deployed with the CDK looks like this.

Problem

Now that we’ve set up the basic infrastructure, we’re going to create problems for ourselves. Suppose Role B no longer needs to access the S3-Bucket and because of that we remove it from the policy. The new policy statement in the importing stack looks like this with role_b commented out:

test_bucket.add_to_resource_policy(
    iam.PolicyStatement(
        actions=["s3:GetObject"],
        principals=[
            role_a,
            # role_b
        ],
        resources=[
            test_bucket.arn_for_objects("*"),
            test_bucket.bucket_arn
        ]
    )
)

In theory this is great and a cdk synth won’t show any errors. The problem arises, when we try to deploy the change:

As you can see, we’re getting this error message:

Export export:ExportsOutputFnGetAttexportingroleb66286D65ArnE09A9A52 cannot be deleted as it is in use by import

Now that’s strange. We’ve just removed that dependency, so why are we getting an error here? Well, if we take a look at the deployment process again, we can see, that the CDK still recognizes a dependency between both stacks and deploys the exporting stack first and the importing stack last. This is correct as we still have a second dependency with role_a here.

The problem is, that the CDK noticed, that the importing stack no longer needs the import value for the role arn of role_b. As a result of which, the CDK has removed the export from the exporting stack. So the template that the CDK generates for the exporting stack no longer contains the export.

Since the CDK deploys the exporting stack first, CloudFormation tries to remove the export before the importing stack has removed the Fn::ImportValue call and that’s not permitted.

Solution

To solve this problem, we can create the output with the export for the exporting stack manually. We just need to make sure that our ids and values match those of the existing stack and we can see and extract those from the CloudFormation console.

Adding a new Output to the exporting stack is easy as well. Note, that I used the override_logical_id function to explicitly set the logical id of the output to the one CDK has generated for us.

compat_output = core.CfnOutput(
    self,
    id="will-be-overwritten",
    value=f"arn:aws:iam::{core.Aws.ACCOUNT_ID}:role/export-exportingroleb66286D65-CZGEAEVHHA32",
    export_name="export:ExportsOutputFnGetAttexportingroleb66286D65ArnE09A9A52"
)
compat_output.override_logical_id("ExportsOutputFnGetAttexportingroleb66286D65ArnE09A9A52")

If we now deploy our changes after the explicit export has been added, we can see that the deployment succeeds.

Now we’re in a position, where the importing stack no longer references the export from the exporting stack, but the exporting stack still contains the export because we explicitly defined it. Since we no longer need it, we can remove the explicit export from the code and deploy again. Afterwards we should be in a state where there is exactly one export and one import value.

Conclusion

In this post I’ve shown you one of the chicken and egg problems with cross-stack-depedencies in CloudFormation. The CDK makes cross-stack-dependencies a lot easier and I really like that it aims to abstract away dealing with outputs/exports and import value statements. It mostly does a great job at that, but it’s a leaky abstraction (just like almost any abstraction we build). Now you know how to work around those leaks.

You can find the code and everything I referenced here on Github

Thank you for investing so much of your valuable time in reading this till the end - if you have feedback, questions or suggestions feel free to reach out to me on Twitter or any of the other social media channels mentioned in my bio below.