November 8th, 2024

Three Ways to Simplify CI/CD Pipelines on GitHub Actions

Introduction

In our recent engagement, our team crafted reusable and extensible pipelines designed to enhance consistency, improve readability, and drive developer productivity. In this blog, we share our insights to help you elevate your own pipeline efficiency. Continuous Integration and Continuous Deployment (CI/CD) pipelines are critical for ensuring efficient and reliable code deployment. As projects grow, these pipelines can become increasingly complex, making them difficult to manage and extend.

By adopting the following three approaches, you can streamline pipeline management, lower operational overhead, and boost productivity:

  1. Modularize CI/CD Pipelines using reusable workflows
  2. Convert Bicep outputs to JSON for better integration
  3. Utilize Bicep outputs in post-infrastructure deployment steps

Let’s dive into each of these approaches with detailed explanations and code examples.

1. Modularize CI/CD Pipelines Using Reusable Workflow Pattern

GitHub Actions supports reusable workflows, allowing one workflow to call another. By modularizing your CI/CD pipelines using this pattern, you enhance maintainability and readability across your workflows.

Benefits of Modularization

  • Maintainability: Update individual components without affecting the entire pipeline.
  • Reusability: Share common workflows across multiple projects or repositories.
  • Clarity: Simplify complex pipelines by breaking them into smaller, manageable parts.

a. Modularized Continuous Integration (CI) Pipeline

Here’s how to create a modularized CI pipeline using GitHub Actions to validate code on pull requests or merges to protected branches.

Create a ci.yaml that calls other reusable workflows for its linting and test tasks:

name: Continuous Integration
on:
  workflow_call: # allow to be called from other workflows
  workflow_dispatch: # allow triggering manually
  pull_request: # on any PR
jobs:
  linting:
      uses: ./.github/workflows/linting.yaml
  test_dotnet:
    uses: ./.github/workflows/test_dotnet.yaml
  test_python:
    uses: ./.github/workflows/test_python.yaml

Each job references a separate re-usable workflow:

  • linting.yaml: Contains steps for code linting.
  • test_dotnet.yaml: Contains steps for .NET unit tests.
  • test_python.yaml: Contains steps for Python unit tests.

Reference linting workflow linting.yaml:

name: Linting
on:
  workflow_call: # allow to be called from other workflows
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run Linting
        run: |
          # Add linting commands here
          echo "Linting code..."

The test_{language}.yaml workflows can be established in a similar way to call setup the language-specific toolchains and call dotnet test or pytest.

b. Modularized Continuous Deployment (CD) Pipeline

Similarly, create a modularized CD pipeline to deploy the application. This workflow is divided into several jobs, each responsible for a specific deployment task such as parsing environment information, deploying infrastructure, or pushing Docker images.

This modular approach ensures that each deployment step in the CD pipeline is self-contained and can be reused or triggered manually for a partial deployment if required.

Reference cd.yaml:

name: Continuous Deployment

on:
  workflow_dispatch: # allow triggering manually
    inputs:
      # manually triggered runs have GITHUB_REF_NAME as a branch, not a tag so we need to manually ask for the environment
      environment_name:
        description: "GitHub deployment environment to pull variables and secrets from"
        required: true
        type: environment
  push:
    branches:
      # pushing to a branch called releases/foo will automatically trigger a deployment using the 'foo' environment (as configured in GitHub Environments)
      - releases/**

jobs:
  # Run CI before deployment
  ci:
    uses: ./.github/workflows/ci.yaml

  detect_env:
    needs: [ci] # don't proceed with environment deployment unless CI succeeded
    runs-on: ubuntu-latest
    outputs:
      # environment name is the name of the GitHub Deployment Environment to reference
      environment_name: ${{ steps.parse_ref.outputs.environment_name }}
      # build_version will be used to tag docker images/packages/artifacts
      build_version: ${{ steps.parse_ref.outputs.build_version }}
    steps:
      # Obtain environment name from Git tag/branch when possible
      - id: parse_ref
        name: Parse Environment from Git Reference
        run: |
          build_version="${{ github.sha }}"
          if [ ! -z "${{ inputs.environment_name }}" ];then
            # user specified environment; do not auto-detect
            environment_name="${{ inputs.environment_name }}"
            echo "Manually specified environment '$environment_name', setting build version to SHA '$build_version'"
          else
            # no environment specified; use the current branch/tag name without the 'releases/' prefix
            environment_name="${{ github.ref_name }}"
            environment_name="${environment_name##releases/}"
          fi

          echo "Using environment '$environment_name' with build version '$build_version' from ref ${{ github.ref_name }}"
          echo "environment_name=$environment_name" >> $GITHUB_OUTPUT
          echo "build_version=$build_version" >> $GITHUB_OUTPUT

  deploy_infra:
    needs: [detect_env]
    uses: ./.github/workflows/deploy_infra.yaml
    secrets: inherit
    with:
      environment_name: ${{ needs.detect_env.outputs.environment_name }}
      build_version: ${{ needs.detect_env.outputs.build_version }}

  docker_build_and_push:
    permissions:
      contents: read
      packages: write
    needs: [deploy_infra]
    uses: ./.github/workflows/docker_build_and_push.yaml
    secrets: inherit
    with:
      environment_name: ${{ needs.detect_env.outputs.environment_name }}
      build_version: ${{ needs.detect_env.outputs.build_version }}

  deploy_app:
    needs: [docker_build_and_push]
    uses: ./.github/workflows/deploy_app.yaml
    secrets: inherit
    with:
      environment_name: ${{ needs.detect_env.outputs.environment_name }}

Each job in the CD pipeline focuses on a specific task:

  • ci: Runs the CI workflow to ensure code quality before deployment.
  • detect_env: Determines the deployment environment, either use-specified or from the Git branch/tag name.
  • deploy_infra: Deploys infrastructure using Bicep templates.
  • docker_build_and_push: Builds and pushes Docker images.
  • deploy_app: Deploys the application to the target environment.

2. Convert Bicep Outputs to JSON for Better Integration

When deploying Azure infrastructure with Bicep, you often need to pass outputs (e.g. resource names, endpoint URIs, etc) to subsequent steps or workflows to configure the newly deployed infrastructure.

By capturing Bicep outputs and storing them as JSON artifacts, you can easily pass infrastructure details to other jobs or workflows. This method will streamline your CD pipeline.

Reference infrastructure deployment workflow

Below is an easy-to-understand code example demonstrating how to deploy infrastructure using Bicep and capture its outputs for better integration.

Contents of deploy_infra.yaml:

name: Deploy infrastructure

on:
  workflow_call: # allow to be called from other workflows
    inputs:
      environment_name:
        required: true
        type: string
      build_version:
        required: true
        type: string
  workflow_dispatch: # allow triggering manually
    inputs:
      environment_name:
        description: "GitHub deployment environment to pull variables and secrets from"
        required: true
        type: environment
      build_version:
        description: "Build version (usually commit SHA) to tag images with"
        required: true
        type: string

jobs:
  iac:
    runs-on: ubuntu-latest
    # pull SUBSCRIPTION_ID, RESOURCE_GROUP variables and AZURE_CREDENTIALS json secret in the GitHub Environment
    # see https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/using-environments-for-deployment
    environment: ${{ inputs.environment_name }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Login to Azure
        uses: azure/login@v2
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}

      # Map in all variables that start with BICEP_ from GitHub deployment environment as environment variables
      - name: Set env vars from vars context JSON
        id: ghenv-to-envvar
        env:
          # https://github.com/actions/runner/issues/1656#issuecomment-1030077729
          VARS_JSON: ${{ toJSON(vars) }}
        run: |
          echo $VARS_JSON | jq -r 'to_entries | .[] | select(.key | startswith("BICEP_")) | "\(.key)=\(.value)"' >> $GITHUB_ENV

      - id: bicep-deploy
        name: "Bicep deployment"
        uses: azure/arm-deploy@v1
        env:
          # Map in any other variables that aren't directly set in GitHub environment
          BICEP_ENVIRONMENT_NAME: ${{ inputs.environment_name }}
          BICEP_RESOURCE_GROUP: ${{ vars.RESOURCE_GROUP }}
        with:
          subscriptionId: ${{ vars.SUBSCRIPTION_ID }}
          resourceGroupName: ${{ vars.RESOURCE_GROUP }}
          template: deployment/iac/main.bicep
          parameters: deployment/iac/main.bicepparam

      - name: Convert Bicep outputs to infra-outputs.json
        env:
          ENV_VARS_JSON: ${{ toJSON(steps.bicep-deploy.outputs) }}
        run: |
          # non-string outputs are stringified by GitHub Actions - parse the values as JSON when possible
          jq -re 'to_entries | map({(.key): (.value | fromjson? // .)}) | add' <<< "$ENV_VARS_JSON" > infra-outputs.json

      - name: Upload infra-outputs.json
        uses: actions/upload-artifact@v4
        with:
          name: infra-outputs-json
          path: infra-outputs.json
          if-no-files-found: error
          overwrite: true

      # Make Bicep outputs available as prefixed env vars
      # In other workflows, remember to download the artifact above first
      - name: JSON to variables
        shell: bash
        run: |
          cat infra-outputs.json | jq -r '[paths(scalars) as $path | {"key": $path | map(if type == "number" then tostring else . end) | join("_"), "value": getpath($path)}] | map("\(.key)=\(.value)") | .[]' | while read line;do
            echo "${{ inputs.prefix }}${line}" >> $GITHUB_ENV
          done

      # now, a Bicep output 'foo.bar' can be referenced as '$INFRA_foo_bar' in subsequent steps
      # ...

Explanation of Key Steps

  • Set env vars from vars context JSON: Extracts configured GitHub environment variables that start with BICEP_ to be used as shell environment variables in the Bicep deployment (as variables configured in GitHub environments for deployment are not exposed to the shell by default).
  • Bicep deployment: Uses the azure/arm-deploy@v1 action to deploy the infrastructure defined in the Bicep templates.
  • Convert Bicep outputs to infra-outputs.json: Converts the outputs from the Bicep deployment to a infra-outputs.json file.
  • Upload infra-outputs.json: Uploads the infra-outputs.json file as an workflow artifact, making it accessible to other reusable workflows in the CD pipeline.
  • JSON to variables: reads the infra-outputs.json file and exposes the outputs as shell environment variables (prefixed by $INFRA_), making all the Bicep outputs accessible to downstream job steps.

3. Utilize Bicep Outputs in Post-Infrastructure Deployment Steps

Building upon the previous point, you can now use the Bicep outputs stored as JSON in subsequent deployment steps. This approach simplifies post-infrastructure deployments by providing easy access to resource details like names, connection strings, and endpoints.

Example: Deploying an Application Using Bicep Outputs

Below is an example of how to use the Bicep outputs in a deployment workflow to deploy an application to Azure App Service.

Reference application deployment workflow (deploy_app.yaml):

name: Deploy Application to Azure App Service

on:
  workflow_call: # allow to be called from other workflows
    inputs:
      environment_name:
        required: true
        type: string
  workflow_dispatch: # allow triggering manually
    inputs:
      environment_name:
        description: "GitHub deployment environment to pull variables and secrets from"
        required: true
        type: environment
      # if triggering manually, users provide the run ID of a previous deployment run to grab infra outputs from
      deployment_run_id:
        description: "Run ID of the deployment to use infrastructure variables from"
        required: true
        type: number

jobs:
  deploy_app:
    runs-on: ubuntu-latest
    environment: ${{ inputs.environment_name }}
    steps:
      # Download the Bicep outputs artifact
      - name: Download infra-outputs.json
        uses: actions/download-artifact@v4
        with:
          name: infra-outputs-json
          run-id: ${{ inputs.deployment_run_id || github.run_id }}

    # Expose bicep outputs as shell variables with a $INFRA_ prefix
    - name: JSON to variables
      shell: bash
      run: |
        cat infra-outputs.json | jq -r '[paths(scalars) as $path | {"key": $path | map(if type == "number" then tostring else . end) | join("_"), "value": getpath($path)}] | map("\(.key)=\(.value)") | .[]' | while read line;do
          echo "${{ inputs.prefix }}${line}" >> $GITHUB_ENV
        done

      - name: Azure CLI Login
        uses: azure/login@v2
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}

      # Use the $INFRA_* environment variables to get resource names output by Bicep
      - name: Deploy Application to App Service
        run: |
          APP_SERVICE_NAME="$INFRA_names_appService"
          RESOURCE_GROUP="$INFRA_resourceGroup"

          echo "Deploying to App Service: $APP_SERVICE_NAME in Resource Group: $RESOURCE_GROUP"

          # Deploy application code to the App Service
          az webapp deploy \
            --name "$APP_SERVICE_NAME" \
            --resource-group "$RESOURCE_GROUP" \
            --src-path ./app \
            --type zip

Explanation of Key Steps

  • Download infra-outputs.json: Downloads the infra-outputs.json artifact containing the Bicep outputs from the infrastructure deployment.
  • JSON to Variables: Converts the JSON file into environment variables prefixed with $INFRA_ for easy access.
  • Azure CLI Login: Authenticates with Azure using provided credentials.
  • Deploy Application to App Service:
    • Retrieves the App Service name and resource group from the environment variables set from Bicep outputs. It assumes the Bicep has an output names.appService, which becomes available as $INFRA_names_appService in the CI shell.
    • Uses the Azure CLI to deploy the application code to the specified App Service.

Conclusion

Modularizing pipelines with reusable workflows and converting Bicep outputs to JSON enables you to create developer-friendly CI/CD pipelines on Azure. Utilizing these outputs in subsequent deployment steps not only simplifies the process but also enhances the overall efficiency of your pipeline.

Implementing these strategies enables your development team to:

  • Focus on Code Quality: Spend more time improving the application rather than managing complex pipelines.
  • Increase Productivity: Reduce the overhead associated with pipeline maintenance.
  • Promote Consistency: Ensure consistent deployment processes across different environments and projects.

Start adopting these practices today to streamline your CI/CD pipelines and accelerate your development workflow.

*The featured image was generated using ChatGPT’s DALL-E tool.

Author

Stewart Adam
Sr. Software Engineer