The case of the missing bucket notifications
This content is more than 4 years old and the cloud moves fast so some information may be slightly out of date.
The case of the missing bucket notifications
A few days ago I was trying to do something quite simple. I wanted to send S3 Put-Events to multiple Lambda functions for processing. This is a pretty common pattern. To implement it you have to use an architecture such as the one you can see in the title image above.
This is because S3 has a limit on the event handlers (notification targets) per event type of exactly one. That’s not a lot. To work around that it’s common to use SNS as an intermediary to fan out the event notifications to multiple targets.
I had done the same as part of a larger infrastructure but for some reason my lambdas weren’t getting any events from S3.
I tried a lot to figure out the issue, but everything seemed to be configured correctly.
- The deployment pipeline was deploying my CloudFormation template correctly
- The notification was configured correctly in the S3 GUI.
- Even after changing the name of the notification in the GUI the S3-check still reported it as being configured correctly
- Publishing other events to SNS triggered my Lambdas
So it seemed like the problem was the communication between S3 and SNS. That was bad, because there’s not a lot of logs (none) you can look at.
I built the whole thing manually in my personal account via the GUI and everything seemed fine - all notifications were being triggered as expected. That annoyed me a little to say the least so I finally decided to extract the involved parts from the infrastructure and create a minimal CloudFormation template for it.
Resources:
EncryptionKey:
Type: AWS::KMS::Key
Properties:
Description: KMS Key
KeyPolicy:
# Allow access from our AWS Account
Version: "2012-10-17"
Id: "kms-key"
Statement:
- Sid: "Enable IAM User Permissions"
Effect: Allow
Principal:
AWS:
Fn::Join:
- ""
- - "arn:aws:iam::"
- !Ref AWS::AccountId
- ":root"
Action: "kms:*"
Resource: "*"
S3BucketFileStorageBucket:
DependsOn: NewFileNotifications
Type: AWS::S3::Bucket
Properties:
BucketName: randombucketname0928439843293874
# Enable Encryption
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: AES256
# Block all public access
AccessControl: Private
NotificationConfiguration:
TopicConfigurations:
- Event: "s3:ObjectCreated:*"
Topic: !Ref NewFileNotifications
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
NewFileNotifications:
Type: AWS::SNS::Topic
Properties:
KmsMasterKeyId: !Ref EncryptionKey
TopicName: NewFileNotifications
NewFileNotificationsTopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Id: AllowPublishFromS3
Version: '2012-10-17'
Statement:
- Sid: Statement-id
Effect: Allow
Principal:
AWS: "*"
Action: sns:Publish
Resource:
Ref: NewFileNotifications
Condition:
ArnLike:
aws:SourceArn: 'arn:aws:s3:::randombucketname0928439843293874'
Topics:
- Ref: NewFileNotifications
Minimal insofar as CloudFormation templates can be minimal. I deployed it and saw the same behaviour as in the real environment, which was good… and bad. It meant I had done something wrong and couldn’t blame AWS.
You might have noticed something I hadn’t mentioned yet - the bucket and topic were encrypted via KMS.
Turns out looking at my own code would have helped - my comment mentions Allow access from our AWS Account but the statement Id says Enable IAM User Permissions.
Past me had lied and present me believed him and assumed the account hat access to the key.
This section allows all IAM-Users to access the key:
EncryptionKey:
Type: AWS::KMS::Key
Properties:
Description: KMS Key
KeyPolicy:
# Allow access from our AWS Account
Version: "2012-10-17"
Id: "kms-key"
Statement:
- Sid: "Enable IAM User Permissions"
Effect: Allow
Principal:
AWS:
Fn::Join:
- ""
- - "arn:aws:iam::"
- !Ref AWS::AccountId
- ":root"
Action: "kms:*"
Resource: "*"
That’s why the checks S3 does to verify the connection to SNS worked fine when I executed them from the GUI and during CloudFormation deployment. Both are triggered via real Users - in the case of the GUI my own user and the technical user for the pipeline.
After finding out what I had to look for, I quickly found this AWS blog, which describes how to do it properly. Once you know what’s wrong, it seems like the whole internet tells you how to do it properly and makes you feel like an idiot for not knowing it.
After some more research I changed the encryption key configuration to this:
Resources:
EncryptionKey:
Type: AWS::KMS::Key
Properties:
Description: KMS Key
KeyPolicy:
# Allow access from our AWS Account
Version: "2012-10-17"
Id: "kms-key"
Statement:
- Sid: "Enable IAM User Permissions"
Effect: Allow
Principal:
AWS:
Fn::Join:
- ""
- - "arn:aws:iam::"
- !Ref AWS::AccountId
- ":root"
Action: "kms:*"
Resource: "*"
- Sid: "Allow S3 to encrypt"
Effect: Allow
Principal:
Service: s3.amazonaws.com
Action:
- "kms:GenerateDataKey*"
- "kms:Decrypt"
Resource: "*"
- Sid: "Allow SNS to encrypt"
Effect: Allow
Principal:
Service: sns.amazonaws.com
Action:
- "kms:GenerateDataKey*"
- "kms:Decrypt"
Resource: "*"
It allows SNS and S3 to use the KMS Key to encrypt messages.
Summary
If S3 notifications to SNS fail silently, check if you have encryption enabled and make sure that SNS and S3 have access to the key. Out of shame I won’t disclose how long it took me to figure this out ;-)
