AWS Access Management uncovered: Where do all those 403s come from?
In this blog post, I would like to show you the various types of AWS permission management capabilities to generate a better understanding, where access denied API errors (403) may arise from and what ample options there are to grant permissions. One should be familiar with IAM policies to get the most out of the blog.
Motivation
As an AWS authorized instructor champion I often come to point during the training where I am talking about IAM policy evaluation on AWS. You may be aware of identity based policies (like IAM role attached) policies, or of resourced based policies (like S3 bucket policies). You may be even aware of this detailed policy evaluation picture, that explains common scenarios for access denied errors on AWS. Unfortunately, this is neither complete, nor centrally documented. Therefore I decided to write a blog post as a reference on how evaluation logic works in general, and what additional pieces are hidden somewhere inside the documentation (or are not documented at all).
Two major concepts/types of policies
To start with, the phrase policy stands for any kind of managing access throughout this blog, not scoped only to IAM policies. Let me further introduce two non-official types of policies, or to be more precise, ways of managing access to the AWS API. You may be aware, that there is a slight difference between API methods (around 13.5 k as of today) and IAM permissions (around 14.6k as of today), see here for a good reference. There are API methods, that do not have a backing IAM permission and vice versa, and there are edge cases, where it is a one-to-many mapping between both. The focus of the blog is to elaborate API access denied return values, no to explain IAM.
The first cluster of policies I call generously. That is, all policies within this category allow for API access in general and there is only a single allow statement needed within a single type of policy. The most common example may be an IAM user policy within the same AWS account. Below is a list of different types.
The second cluster of policies I call bounding. That is, whenever a policy of this kind is in place, there must be explicit allows contained in every single policy, so that the overall access is bound to the intersection of all permissions in scope.
The general evaluation logic for API access now can be rephrased a bit more abstract like this:
-
Combine all occurring statements in all policies in place. If there is any deny anywhere, this will lead to final
access denieddecision. -
For every bounding policy in place, check if there is an explicit allow in every policy. If this is false, the final decision is
access denied. -
Check, if there is at least one generously policy in place with an explicit allow. If this is false, the final decision is
access denied. Otherwise you are granted access.
This cheat image illustrates this general logic:
No that we understand the general concept, let’s have a look which types fall under which category in the following tables. You may be familiar with the first entries in every table, maybe some entries you never heard of, and maybe there is still something missing (but you may read the end of the blog post below as well). At least, there are more included than in the referenced picture in the motivation section:
| Generously | Detail |
|---|---|
| IAM User policy | Directly attached (inline) policies to IAM user |
| IAM Group policy | Directly attached (inline) policies to IAM group |
| IAM role policy | Directly attached (inline) policies to IAM roles |
| IAM policy | Standalone IAM policy that can be attached to other entities |
| AWS SSO (now: AWS Identity Center) permission set | IAM policies can now be referenced, but SSO permission sets can still be used. Those will be converted to IAM roles and IAM role policies by AWS Identity Center within the target account, but can only be modified via AWS Identity Center |
| Resource-based policy | Only within the same AWS account, resource-based policies are sufficient to grant access. Though there are valid use cases, where you can use them only with conditional denies to protect certain resources, e.g. enforcing encryption on bucket level wih an explicit KMS key, which default-encryption is no capable of. See below for a more detailed list. |
| SSM agent default host management policy | IAM policy that is referenced within SSM and applied as default permission to EC2 instances similar to EC2 instance profiles (but different authentication) |
| S3 access control list | Kind of deprecated (and most likely at some point in the future removed by AWS), old-fashioned way to grant access to single objects from S3 |
| Account root user | Except for S3 object lock in compliance mode and MFA devices I am not aware of any other account internal way to restrict the root user (i.e., with methods within the target account itself only) |
| AWS RAM | Sharing resources with the use of resource access manager falls a bit in both categories, as you at first give general transparent access for your resources to another account. But in the same moment, one must have permissions within the target account to view/use those resources. |
These are (for me, as of today) the most important bounding policies:
| Bounding | Detail |
|---|---|
| Service Control Policy (SCP) | A very common feature from AWs Organizations, where you must put explicit allow statements on every layer (OU, Account). AWS will add a default “allow all” SCP if you do not give tailored ones. The frame “inheritance” in the documentation is a bit misleading or unintuitive, as only deny statements within SCPs attached to parent OUs will be inherited throughout the different layers, while allow statements have to be given explicitly on every single layer. |
| IAM role permission boundary | See for example a blog post by Maurice for a common scenario of how to use those. |
| IAM role session policy | Almost the same as role permission boundaries but with a temporary scope (i.e., per session) |
| VPC endpoint policy | Other than stated in the documentation, those are not working like usual resource-based policies, that directly allow API actions. If you chose not to enable those, AWS will apply a default “allow all” policy |
| S3 public access block (bucket and account level) | One can chose to enable (or now for new buckets: chose to not disable) access block settings on an account, access point, or bucket level. That will deny access all API calls trying to enable public ACLs or allow-all bucket policies. |
| S3 Object Ownership | See another blog from us on this edge case for example, if you are (still) using ACLs and manage some cross-account access to and from the objects. |
| IAM Password policy | This will essentially deny console password setting for IAM users only, if the password does not stick to the pattern enforced. |
| AWS opt-in for regions | Very edgy case, again, most commonly used for standalone AWS accounts to restrict usage of certain regions. It is way more common to use Organizations SCPs for that, but until now, this is still possible within AWS Organizations as well. |
| CloudFormation Stack policy | Feature from CloudFormation itself that works a bit like resource-based policies for the resources defined in the stack. It is designed to protect stack resources from being accidentally deleted/modified by automation during stack updates. But this will not protect the real resources behind, only the logical resources of the stack (so one could easily delete a server from the console, that is protected by a stack policy). Stack deletion protection works the same way but scopes the complete stack as logical resource. |
| Resource-based policy | Only for cross-account scenarios (i.e., the identity is not the account the resource resides in), resource-based policies work like bounding policies as the identity must have according permissions in the source account as well. See below for a more detailed list. |
| KMS key policies | KMS key policies are behaving as strict bounding policies as there is no AWS provided default if you forget to set a policy. Instead, one has to provide one per key always, and those are not working like usual resource-based policies (as they are bounding within the Account as well). KMS policies and grants would be worth another blog post, we will not go into detail on those here. |
Resource-based policies
There are plenty of AWS services that allow the use of resource-based policies. As of June 2023, there are at least the following commonly used services, that support those. See here for a curated yet uncomfortable to use list of AWS services, that may support resource-based policies and here on how to (mis-)use them to find out another account’s IAM role and user names. One can have the following quite simple use case in mind, where one wants to interact via and API gateway and lambda integration with an S3 bucket with KMS encrypted objects in another (central) account, e.g., a simple web storage app. Now there are already at least 4 different policies involved, where of two (Lambda and S3) are usual resource-based policies, that grant access to other services or accounts.
The following are my favorite most common ones:
| Common resource-based policies | Comment |
|---|---|
| S3 bucket | Not to interfere with ACLs, access point policies, S3 Glacier archive policies (which behave different), or with lifecycle policies, which have the scope of moving/deleting data automatically |
| Lambda | One can essentially allow to invoke functions with these |
| SQS | Most common use case here is to share access to queues across AWS accounts |
| SNS | Most common use case here is to share access to topics across AWS accounts |
| ECR repository | Not to interfere with lifecycle policies, which have the scope of deleting data automatically |
| Cloud9 | Cloud9 resource-based policies are not directly configurable, but AWS makes use of the same technology in the background to grant access to the environment and host for authorized identities. |
| API Gateway | One can use this to allow executions on certain HTTP verbs for the built API |
| CodeBuild | |
| AWS Glue | One can share data catalogs (or restrict access to it), that contain the tables meta data |
| Secrets Manager | One can share secrets across accounts that way, but also improve security via conditional denies |
| Event Bridge | |
| OpenSearch | The policy only applies to subdomains and indices of a certain cluster/domain |
| VPC lattice | New service since 2023 - the policies are called auth policies and allow for network access to the services in scope (i.e., applications behind ALB or Lambda) similar to the functionality of API Gateway |
| AWS Backup | Not to interfere with AWS Organizations backup policies, those grant access/protect resources within a backup vault. Like with AWS S3 object lock, one can add a locking on vaults in another way as well. |
| SSM Incident Manager | One can use this in an AWS Organizations context for granting access to Incident Manager response plans and contacts. |
| IAM assume role policy | The most common and mandatory resource-based policy, aka trust policy, where one must state, who can assume this role. Keep in mind, that when adding the ARN for a specific role or user in the principal element, AWS transforms the ARN to a unique principal ID, which can lead to errors when deleting and naming the role/user the same again. |
Further access management capabilities
We left out a few other services, that allow for access management. There exist roughly three categories of those:
- Identity Management
- Delegated authorization
- Built-in IAM
For the first category, i.e., identity management, there exist for example the Amazon Cognito service. It is capable to authenticate (identity pools) and authorize (via IAM roles), so make user management for web apps or some AWS services like CloudWatch public dashboards or AWS Sumerian VR. Some common open-source implementations for this use case do not (yet) exist as AWS managed service. But at least there is the AWS Directory service, where one can choose from a managed LDAP (Simple AD) or a managed Windows active directory.
The second category not only includes delegation via IAM roles to a trusted web identity or other source of trust, but services like Kubernetes on EKS or RDS (and Aurora) to name the most common. Those have an onboard internal identity and access/permission system, but is possible to delegate the authorization part to IAM (i.e., trust an IAM entity) and map this internally to some authorized actions within the application (kubernetes, mysql, …) itself. From a security point of view, it is most times the preferred solution, to delegate trust to a single point of truth. This may be the reason, why AWS constantly improves in this area and has added over time more and more services. For example, one can enforce authorization of the NFS client through IAM with EFS. That way, one can use EFS policies to grant read or read/write access to the files system itself. Another brand-new service (GA in June 2023) you might think of, is AWS Verified access. The scope is more VPN-less network connectivity rather than API access, nevertheless you will find some similar concepts and verbs within the verified access policies, which are evaluated through the Cedar policy language.
This leads us to the third category. There exist AWS home grown services with a built-in IAM, like Amazon Quicksight or AWS Lake Formation. There you have to setup a user and access management in addition to the IAM capabilities. Observe, that Dynamo DB does not fall into this category as there is a full coverage within AWS IAM for all kinds of actions against your tables, down to field level security (fine grained access control). Other service, which fell into this category (or close to) are Athena work groups and Glue data catalog, where you can make use of AWS IAM policies directly.
In all three cases above, usually possible access denied messages are returned from some different system than the AWS API and are therefore not focus of this post here.
Summary
So, the key take away message is, to understand the difference between bounding and generous policies and how AWS evaluates those, in case they exist. As you can see, permission management within AWS can be subtle and ample, just like the AWS in general. And we did not yet touch the art of writing least privileged policies at all. One can spend days and weeks of having fun, combining those concepts above to ensure context-based, fine-grained access controls at a minimum level.
Feel free to reach out to us at any time if you are interested in trainings, for example the Security Engineering class, or looking for a job.
— Felix
