21 Mar '23

Secretless Terraform Deployments

Written by Hendrik Hagen

When deploying AWS infrastructure via Terraform, the usage of long-term IAM credentials is often the go-to method. Even though convenient, long-term credentials and secrets can pose a serious security risk to your organization if leaked.

This post will show you how you can leverage AWS IAM OpenID Connect identity providers in combination with GitHub Actions and identity federation to implement a secretless Terraform deployment pipeline…using Terraform.

Introduction

According to OWASP, the second biggest of the top 10 web application security risks in 2021 were sensitive data exposures (including credential leaks). A Data Breach Investigation Report done by Verizon in 2022 found that 61% of data breaches were due to leaked credentials. Even though the risk of using long-term credentials is generally known, they are still widely used throughout pipelines, servers, laptops, desktops, and many more locations.

In the case of a leak, the impact on business operations, the financial and reputational damages, and the following legal actions could be catastrophic. Take the example of Code Spaces, a company that was forced to shut down after an attacker got hold of leaked credentials and deleted their entire AWS infrastructure including backups.

In order to not become the next Code Spaces or wake up to a hefty bill due to Crypto Mining Hackers, it is important to take credential management seriously, especially in the age of Infrastructure as Code and hyper scalers like AWS. Everyone who has deployed infrastructure to AWS using Terraform knows how easy it is to deploy resources once you have the proper permissions.

One solution to get rid of long-term credentials is to use AWS IAM OIDC identity providers to establish a trust relationship with OIDC-compatible IdPs. Instead of creating AWS IAM users with long-term credentials in your account to access AWS resources, identities can be managed outside AWS by external IdPs. Users managed by these IdPs can then use JWTs to assume IAM roles and retrieve temporary credentials from AWS.

In this post, I would like to show you how you can leverage IAM OIDC identity providers in combination with GitHub as an IdP to build a secretless deployment pipeline for Terraform…with Terraform.

OIDC Workflow

Before jumping into the implementation, I would like to discuss the general workflow when using AWS IAM OIDC identity providers and GitHub Actions and how a secretless connection works. The workflow can be divided into five steps which are shown in the figure below.

OIDC Workflow

  1. Register GitHub as IAM OIDC Identity Provider in AWS and create an IAM role IdP identities are allowed to assume.
  2. GitHub Action requests a signed JWT during the job run from the GitHub OIDC provider.
  3. GitHub OIDC provider issues signed JWT to GitHub Action.
  4. GitHub Action sends JWT and ARN of the IAM role which should be assumed to AWS.
  5. AWS validates JWT and grants short-lived credentials to GitHub Action.

Once the credentials have been granted by AWS, the GitHub Action can use the permissions to access AWS resources. In the case of our example, we will use these credentials to deploy our Terraform configuration.

Project Structure

Before we start setting up our GitHub Action and our Terraform configuration, I would like you to create the project structure. Please create the following files and folders.

├── github-content/
│   └── deployment.yml
│   └── main.tf
├── github.tf
├── aws.tf
├── provider.tf
├── variables.tf

As mentioned above, we will build a Terraform deployment pipeline using Terraform. This might be a bit counter intuitive, but we will be using Terraform to bootstrap our whole infrastructure. To get a bit more practice when working with Terraform and to make this example a bit more challenging, we will not only set up the AWS IAM OIDC identity provider, but also our complete GitHub environment including repository, branches, actions, and secrets using Terraform.

The folder github-content contains the content that we will push to our GitHub repository once it has been created via Terraform. github.tf will contain our Terraform configuration for the GitHub infrastructure while aws.tf will contain our AWS configuration for the OIDC identity provider. provider.tf and variables.tf will contain our Terraform provider configuration and our variables.

Prerequisites

Implement AWS IAM OIDC Identity Provider

As already described above, the OIDC workflow starts with registering GitHub as a valid OIDC identity provider in AWS. By registering GitHub as a trusted IdP, we allow GitHub identities to request temporary credentials and assume IAM roles from our AWS account. Please copy the following code into your aws.tf.


################################################################################
# Get GitHub TLS cert
################################################################################

data "tls_certificate" "github" {
  url = "https://token.actions.githubusercontent.com/.well-known/openid-configuration"
}


################################################################################
# IAM OpenID Connect for GitHub
################################################################################

resource "aws_iam_openid_connect_provider" "this" {
  url = "https://token.actions.githubusercontent.com"

  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.github.certificates[0].sha1_fingerprint]
}

The code snippet above will create a GitHub identity provider in AWS. As we will be working with two Terraform providers - aws and github - we will set up a provider configuration to make sure that the correct providers and versions are downloaded by Terraform. Copy the following code into your provider.tf.


terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">=4.40.0"
    }
    github = {
      source  = "integrations/github"
      version = ">=5.0"
    }
  }
}

provider "github" {}

provider "aws" {}

Before running a Terraform command, you need to set up the credentials for AWS and GitHub provider. For all possible configuration options please visit the official Terraform documentation regarding the AWS Provider and the GitHub Provider. In the case of GitHub we will use an API key that we will export as an environment variable.

export GITHUB_TOKEN=xxxxx

Run terraform init and terraform apply to deploy the Terraform configuration. You should see the OIDC provider appear in the IAM console in AWS.

OIDC Provider

The provider alone won’t be of much use on its own. We also have to create an IAM role that our GitHub Action will be able to assume to get the temporary AWS credentials. Copy the following snippet into your aws.tf as well.


################################################################################
# IAM Role for GitHub
################################################################################

resource "aws_iam_role" "this" {
  name = "github-actions"

  assume_role_policy = data.aws_iam_policy_document.assume_role_policy.json
}


data "aws_iam_policy_document" "assume_role_policy" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    effect  = "Allow"

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.this.arn]
    }

    condition {
      test     = "StringLike"
      variable = "token.actions.githubusercontent.com:sub"
      values = [
        "repo:${var.org_or_user_name}/${var.repository_name}:pull_request",
        "repo:${var.org_or_user_name}/${var.repository_name}:ref:refs/heads/main"
      ]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }
  }
}

resource "aws_iam_role_policy_attachment" "this" {
  role = aws_iam_role.this.name

  policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"
}

The Terraform configuration will create an IAM role which can be assumed by GitHub through our OIDC identity provider. By default, the maximum duration an IAM role can be assumed and therefore the default maximum validity of the AWS credentials is one hour. If you have long-running workflows, you can adjust the session duration via the max_session_duration input. The values have to be between 1 hour and 12 hours.


resource "aws_iam_role" "this" {
  name = "github-actions"

  max_session_duration  = 12
  assume_role_policy    = data.aws_iam_policy_document.assume_role_policy.json
}

The role has been granted administrator access as part of this example.

This is only for demonstration purposes. Please apply the principle of least privilege when working in production environments.

A token.actions.githubusercontent.com:sub condition has also been added to our assume role policy to make sure that only selected organizations or users and only specific repositories will be able to assume the role. This allows us to apply fine-grained access control based on repositories, branches, and much more. As you can see, the organization/user as well as the repository name were parameterized. Please add the two variables to your variables.tf.


variable "org_or_user_name" {
  description = "Name of GitHub Org or User that can assume IAM role"
  type        = string
  default     = "PUT_YOUR_ORG_OR_USER_HERE"
}

## The name of the repository MUST be a name that you currently DO NOT possess! The repository will be created using Terraform.

variable "repository_name" {
  description = "Name of GitHub repository that can assume IAM role"
  type        = string
  default     = "PUT_YOUR_REPO_NAME_HERE"
}

Feel free to work with a terraform.tfvars file to assign values to these variables. You can also assign a default value for simplicity. Please change the default value of both variables and assign your user name or org and select a name for a repository. Rerun terraform apply to create the IAM Role. You should see the role in the AWS console.

IAM Role

IMPORTANT:

The name of the repository MUST be a name that you currently DO NOT possess! We will create this repository via Terraform later on. If you want to implement the IAM OIDC identity provider on its own and use an existing repository, the repository_name can also be a pattern. Patterns can be used to allow multiple repositories to assume the role. An example is shown below, where instead of a specific repository, access is granted to all repositories via an asterix (*).


    condition {
      test     = "StringLike"
      variable = "token.actions.githubusercontent.com:sub"
      values = [
        "repo:${var.org_or_user_name}/*:pull_request",
        "repo:${var.org_or_user_name}/*:ref:refs/heads/main"
      ]
    }

Create GitHub Environment

After having implemented the OIDC identity provider including IAM role, we will continue by creating the GitHub environment. We will set up a brand new repository with two branches - dev and main. We will also create our GitHub Action secrets and push our example content to the dev branch. Once everything has been set up, we will create a Pull Request and Merge our content into main to showcase how our Terraform deployment pipeline behaves. Let’s start by creating our repository and uploading our GitHub Action workflow into our main branch. We will start with the Action. Copy the following snippet into the deployment.yml.


name: Deployment

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

env:
  AWS_REGION : "eu-central-1"

permissions:
  contents: read
  id-token: write

jobs:

  deployment:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v2

    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-region: ${{ env.AWS_REGION }}
        role-to-assume: ${{ secrets.DEPLOYMENT_IAM_ROLE_ARN }}
        role-session-name: OIDCSession

    - name: Terraform Init
      id: init
      run: |
        echo "\`\`\`\n" >> $GITHUB_STEP_SUMMARY
        terraform init -no-color >> $GITHUB_STEP_SUMMARY
        echo "\`\`\`\n" >> $GITHUB_STEP_SUMMARY        

    - name: Terraform Plan (On Pull Request)
      id: plan
      if: github.event_name == 'pull_request'
      run: |
        echo "\`\`\`\n" >> $GITHUB_STEP_SUMMARY
        terraform plan -no-color >> $GITHUB_STEP_SUMMARY
        echo "\`\`\`\n" >> $GITHUB_STEP_SUMMARY        


    - name: Terraform Apply (On Merge)
      id: apply
      if: github.event_name != 'pull_request'
      run: |
        echo "\`\`\`\n" >> $GITHUB_STEP_SUMMARY
        terraform apply -no-color -auto-approve >> $GITHUB_STEP_SUMMARY
        echo "\`\`\`\n" >> $GITHUB_STEP_SUMMARY

This workflow will start by checking out our code, setting up Terraform, and assuming our IAM role via the OIDC identity provider. The ARN of the IAM role will be provided to the workflow via a GitHub Action secret that we will create. Once the setup process has been completed, the workflow will initialize Terraform and perform terraform plan in the case of a Pull Request and terraform apply in the case of a Merge. This allows us to review the changes Terraform intends to perform before the actual deployment.

Next, we will create the actual repository, actions, and secrets using Terraform. Continue by copying the following code into your github.tf.


################################################################################
# Set up GitHub Repository
################################################################################

resource "github_repository" "this" {
  name        = var.repository_name

  auto_init = true
  visibility  = "private"
}

resource "github_actions_secret" "this" {
  repository       = github_repository.this.name
  secret_name      = "DEPLOYMENT_IAM_ROLE_ARN"
  plaintext_value  = aws_iam_role.this.arn
}

resource "github_repository_file" "deployment_yml" {
  repository          = github_repository.this.name
  branch              = "main"
  file                = ".github/workflows/deployment.yml"
  content             = file("${path.module}/github-content/deployment.yml")
  overwrite_on_create = true
}

Execute terraform apply to create the repository and push the deployment.yml to the main branch. As mentioned in the official GitHub Action documentation, workflow files need to be stored in the directory .github/workflows/.

Check your GitHub account to see if everything has been deployed successfully. You will see that a GitHub Action has already run but failed during the Terraform apply step. This is due to the fact, that we currently have no Terraform files to apply. What might be more interesting is the fact, that the step Configure AWS Credentials has been executed successfully. That means, that our OIDC identity federation is working as intended.

Now it’s time to deploy some actual infrastructure. We will create a dev branch and push a simple Terraform configuration to dev. Afterward, we will create a Pull Request from dev to main and look at the behavior of our GitHub Action. By creating the dev branch and pushing the main.tf we simulate a new feature being added. We will start by copying the following code into our github-content/main.tf. The configuration is straightforward and will create a single S3 Bucket that will start with the prefix tecracer-oidc-github-blog.


resource "aws_s3_bucket" "this" {
  bucket_prefix = "tecracer-oidc-github-blog"
}

To set up the dev branch copy the following code into your github.tf.


################################################################################
# Create Dev Branch
################################################################################

resource "github_branch" "dev" {
  repository    = github_repository.this.name
  branch        = "dev"
  source_branch = "main"

  depends_on = [
    github_repository_file.deployment_yml
  ]
}

resource "github_repository_file" "sample_tf_config" {
  repository          = github_repository.this.name
  branch              = github_branch.dev.branch
  file                = "main.tf"
  content             = file("${path.module}/github-content/main.tf")
  overwrite_on_create = true
}

Execute terraform apply to create and deploy the changes. You should now have two branches,dev and main, in your repository.

Main Branch

Dev Branch

Create Pull Request and Merge Changes

Now that we have created the infrastrcture for our deployment pipeline, we will finally deploy some infrastructure using our GitHub Action. Create a Pull Request with main as the base.

Pull request

The GitHub Actions will start to run shortly after.

GitHub Action

Once the Action has finished successfully, you should see the Terraform plan output in the job summary. As configured, Terraform intends to add one S3 Bucket to our AWS account.

Plan Output

To deploy the infrastructure, Merge the Pull Request into main. Following the Merge another GitHub Action will start. This time to deploy our Terraform configuraiton. The job summary will show the successful deployment of our S3 Bucket. Check the AWS Console and you should see that your S3 Bucket has been created.

S3 Bucket

Summary

As you can see it is pretty straight forward to create a secretless deployment pipeline using AWS OIDC identity providers and GitHub Actions. One major advantage of this approach is the fact that we got rid of long-termin credentials. We no longer have to create IAM users and store access keys in GitHub to get access to AWS resources. Temporary credentials are created as needed on the fly for a specific GitHub Action. Therefore, The risk of a security breach due to leaked credentials is reduced.

Furthermore, we are able to configure fine-grained access control through IAM roles conditions. By defining IAM role conditions we can define which principals are allowed to assume said role. In the case of GitHub Actions, this can be done based on patterns or specific repositories, branches, actions and much more. Each project repository could receive its own IAM role with separate permissions for example. You could also restrict the permissions of dev and qa branches. The choice is yours. In the end, the goal is to make your IaC deployments safer and more secure.

I hope you had fun and learned something new while working through this short example. I am looking forward to your feedback and questions. If you want to take a look at the complete example code please visit my Github.

— Hendrik

Similar Posts You Might Enjoy

23 Dec '23

Streamlined Kafka Schema Evolution in AWS using MSK and the Glue Schema Registry

In today’s data-driven world, effective data management is crucial for organizations aiming to make well-informed, data-driven decisions. As the importance of data continues to grow, so does the significance of robust data management practices. This includes the processes of ingesting, storing, organizing, and maintaining the data generated and collected by an organization. Within the realm of data management, schema evolution stands out as one of the most critical aspects. Businesses evolve over time, leading to changes in data and, consequently, changes in corresponding schemas. Even though a schema may be initially defined for your data, evolving business requirements inevitably demand schema modifications. Yet, modifying data structures is no straightforward task, especially when dealing with distributed systems and teams. It’s essential that downstream consumers of the data can seamlessly adapt to new schemas. Coordinating these changes becomes a critical challenge to minimize downtime and prevent production issues. Neglecting robust data management and schema evolution strategies can result in service disruptions, breaking data pipelines, and incurring significant future costs. In the context of Apache Kafka, schema evolution is managed through a schema registry. As producers share data with consumers via Kafka, the schema is stored in this registry. The Schema Registry enhances the reliability, flexibility, and scalability of systems and applications by providing a standardized approach to manage and validate schemas used by both producers and consumers. This blog post will walk you through the steps of utilizing Amazon MSK in combination with AWS Glue Schema Registry and Terraform to build a cross-account streaming pipeline for Kafka, complete with built-in schema evolution. This approach provides a comprehensive solution to address your dynamic and evolving data requirements. - by Hendrik Hagen

30 May '23

Build Terraform CI/CD Pipelines using AWS CodePipeline

When deciding which Infrastructure as Code tool to use for deploying resources in AWS, Terraform is often a favored choice and should therefore be a staple in every DevOps Engineer’s toolbox. While Terraform can increase your team’s performance quite significantly even when used locally, embedding your Terraform workflow in a CI/CD pipeline can boost your organization’s efficiency and deployment reliability even more. By adding automated validation tests, linting as well as security and compliance checks you additionally ensure that your infrastructure adheres to your company’s standards and guidelines. In this blog post, I would like to show you how you can leverage the AWS Code Services CodeCommit, CodeBuild, and CodePipeline in combination with Terraform to build a fully-managed CI/CD pipeline for Terraform. - by Hendrik Hagen

10 May '23

Cross Account Kafka Streaming: Part 1

When discussing high performant real-time event streaming, Apache Kafka is a tool that immediately comes to mind. Optimized for ingesting and transforming real-time streaming data in a reliable and scalable manner, a great number of companies today rely on Apache Kafka to power their mission-critical applications and data analytics pipelines. In this blog series, I would like to show you how you can leverage Amazon MSK and Terraform to set up a fully managed, cross-account Apache Kafka streaming pipeline on AWS. In this first part, we will set up the MSK Kafka cluster and producers. The second part will show you how you can set up distributed Kafka clients in different AWS accounts and communicate with the MSK cluster via AWS VPC Endpoints. - by Hendrik Hagen