Introduction
In our recent engagement, our team crafted reusable and extensible pipelines designed to enhance consistency, improve readability, and drive developer productivity. In this blog, we share our insights to help you elevate your own pipeline efficiency. Continuous Integration and Continuous Deployment (CI/CD) pipelines are critical for ensuring efficient and reliable code deployment. As projects grow, these pipelines can become increasingly complex, making them difficult to manage and extend.
By adopting the following three approaches, you can streamline pipeline management, lower operational overhead, and boost productivity:
- Modularize CI/CD Pipelines using reusable workflows
- Convert Bicep outputs to JSON for better integration
- Utilize Bicep outputs in post-infrastructure deployment steps
Let’s dive into each of these approaches with detailed explanations and code examples.
1. Modularize CI/CD Pipelines Using Reusable Workflow Pattern
GitHub Actions supports reusable workflows, allowing one workflow to call another. By modularizing your CI/CD pipelines using this pattern, you enhance maintainability and readability across your workflows.
Benefits of Modularization
- Maintainability: Update individual components without affecting the entire pipeline.
- Reusability: Share common workflows across multiple projects or repositories.
- Clarity: Simplify complex pipelines by breaking them into smaller, manageable parts.
a. Modularized Continuous Integration (CI) Pipeline
Here’s how to create a modularized CI pipeline using GitHub Actions to validate code on pull requests or merges to protected branches.
Create a ci.yaml
that calls other reusable workflows for its linting and test tasks:
name: Continuous Integration
on:
workflow_call: # allow to be called from other workflows
workflow_dispatch: # allow triggering manually
pull_request: # on any PR
jobs:
linting:
uses: ./.github/workflows/linting.yaml
test_dotnet:
uses: ./.github/workflows/test_dotnet.yaml
test_python:
uses: ./.github/workflows/test_python.yaml
Each job references a separate re-usable workflow:
linting.yaml
: Contains steps for code linting.test_dotnet.yaml
: Contains steps for .NET unit tests.test_python.yaml
: Contains steps for Python unit tests.
Reference linting workflow linting.yaml
:
name: Linting
on:
workflow_call: # allow to be called from other workflows
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Run Linting
run: |
# Add linting commands here
echo "Linting code..."
The test_{language}.yaml
workflows can be established in a similar way to call setup the language-specific toolchains and call dotnet test
or pytest
.
b. Modularized Continuous Deployment (CD) Pipeline
Similarly, create a modularized CD pipeline to deploy the application. This workflow is divided into several jobs, each responsible for a specific deployment task such as parsing environment information, deploying infrastructure, or pushing Docker images.
This modular approach ensures that each deployment step in the CD pipeline is self-contained and can be reused or triggered manually for a partial deployment if required.
Reference cd.yaml
:
name: Continuous Deployment
on:
workflow_dispatch: # allow triggering manually
inputs:
# manually triggered runs have GITHUB_REF_NAME as a branch, not a tag so we need to manually ask for the environment
environment_name:
description: "GitHub deployment environment to pull variables and secrets from"
required: true
type: environment
push:
branches:
# pushing to a branch called releases/foo will automatically trigger a deployment using the 'foo' environment (as configured in GitHub Environments)
- releases/**
jobs:
# Run CI before deployment
ci:
uses: ./.github/workflows/ci.yaml
detect_env:
needs: [ci] # don't proceed with environment deployment unless CI succeeded
runs-on: ubuntu-latest
outputs:
# environment name is the name of the GitHub Deployment Environment to reference
environment_name: ${{ steps.parse_ref.outputs.environment_name }}
# build_version will be used to tag docker images/packages/artifacts
build_version: ${{ steps.parse_ref.outputs.build_version }}
steps:
# Obtain environment name from Git tag/branch when possible
- id: parse_ref
name: Parse Environment from Git Reference
run: |
build_version="${{ github.sha }}"
if [ ! -z "${{ inputs.environment_name }}" ];then
# user specified environment; do not auto-detect
environment_name="${{ inputs.environment_name }}"
echo "Manually specified environment '$environment_name', setting build version to SHA '$build_version'"
else
# no environment specified; use the current branch/tag name without the 'releases/' prefix
environment_name="${{ github.ref_name }}"
environment_name="${environment_name##releases/}"
fi
echo "Using environment '$environment_name' with build version '$build_version' from ref ${{ github.ref_name }}"
echo "environment_name=$environment_name" >> $GITHUB_OUTPUT
echo "build_version=$build_version" >> $GITHUB_OUTPUT
deploy_infra:
needs: [detect_env]
uses: ./.github/workflows/deploy_infra.yaml
secrets: inherit
with:
environment_name: ${{ needs.detect_env.outputs.environment_name }}
build_version: ${{ needs.detect_env.outputs.build_version }}
docker_build_and_push:
permissions:
contents: read
packages: write
needs: [deploy_infra]
uses: ./.github/workflows/docker_build_and_push.yaml
secrets: inherit
with:
environment_name: ${{ needs.detect_env.outputs.environment_name }}
build_version: ${{ needs.detect_env.outputs.build_version }}
deploy_app:
needs: [docker_build_and_push]
uses: ./.github/workflows/deploy_app.yaml
secrets: inherit
with:
environment_name: ${{ needs.detect_env.outputs.environment_name }}
Each job in the CD pipeline focuses on a specific task:
ci
: Runs the CI workflow to ensure code quality before deployment.detect_env
: Determines the deployment environment, either use-specified or from the Git branch/tag name.deploy_infra
: Deploys infrastructure using Bicep templates.docker_build_and_push
: Builds and pushes Docker images.deploy_app
: Deploys the application to the target environment.
2. Convert Bicep Outputs to JSON for Better Integration
When deploying Azure infrastructure with Bicep, you often need to pass outputs (e.g. resource names, endpoint URIs, etc) to subsequent steps or workflows to configure the newly deployed infrastructure.
By capturing Bicep outputs and storing them as JSON artifacts, you can easily pass infrastructure details to other jobs or workflows. This method will streamline your CD pipeline.
Reference infrastructure deployment workflow
Below is an easy-to-understand code example demonstrating how to deploy infrastructure using Bicep and capture its outputs for better integration.
Contents of deploy_infra.yaml
:
name: Deploy infrastructure
on:
workflow_call: # allow to be called from other workflows
inputs:
environment_name:
required: true
type: string
build_version:
required: true
type: string
workflow_dispatch: # allow triggering manually
inputs:
environment_name:
description: "GitHub deployment environment to pull variables and secrets from"
required: true
type: environment
build_version:
description: "Build version (usually commit SHA) to tag images with"
required: true
type: string
jobs:
iac:
runs-on: ubuntu-latest
# pull SUBSCRIPTION_ID, RESOURCE_GROUP variables and AZURE_CREDENTIALS json secret in the GitHub Environment
# see https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/using-environments-for-deployment
environment: ${{ inputs.environment_name }}
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Login to Azure
uses: azure/login@v2
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
# Map in all variables that start with BICEP_ from GitHub deployment environment as environment variables
- name: Set env vars from vars context JSON
id: ghenv-to-envvar
env:
# https://github.com/actions/runner/issues/1656#issuecomment-1030077729
VARS_JSON: ${{ toJSON(vars) }}
run: |
echo $VARS_JSON | jq -r 'to_entries | .[] | select(.key | startswith("BICEP_")) | "\(.key)=\(.value)"' >> $GITHUB_ENV
- id: bicep-deploy
name: "Bicep deployment"
uses: azure/arm-deploy@v1
env:
# Map in any other variables that aren't directly set in GitHub environment
BICEP_ENVIRONMENT_NAME: ${{ inputs.environment_name }}
BICEP_RESOURCE_GROUP: ${{ vars.RESOURCE_GROUP }}
with:
subscriptionId: ${{ vars.SUBSCRIPTION_ID }}
resourceGroupName: ${{ vars.RESOURCE_GROUP }}
template: deployment/iac/main.bicep
parameters: deployment/iac/main.bicepparam
- name: Convert Bicep outputs to infra-outputs.json
env:
ENV_VARS_JSON: ${{ toJSON(steps.bicep-deploy.outputs) }}
run: |
# non-string outputs are stringified by GitHub Actions - parse the values as JSON when possible
jq -re 'to_entries | map({(.key): (.value | fromjson? // .)}) | add' <<< "$ENV_VARS_JSON" > infra-outputs.json
- name: Upload infra-outputs.json
uses: actions/upload-artifact@v4
with:
name: infra-outputs-json
path: infra-outputs.json
if-no-files-found: error
overwrite: true
# Make Bicep outputs available as prefixed env vars
# In other workflows, remember to download the artifact above first
- name: JSON to variables
shell: bash
run: |
cat infra-outputs.json | jq -r '[paths(scalars) as $path | {"key": $path | map(if type == "number" then tostring else . end) | join("_"), "value": getpath($path)}] | map("\(.key)=\(.value)") | .[]' | while read line;do
echo "${{ inputs.prefix }}${line}" >> $GITHUB_ENV
done
# now, a Bicep output 'foo.bar' can be referenced as '$INFRA_foo_bar' in subsequent steps
# ...
Explanation of Key Steps
- Set env vars from vars context JSON: Extracts configured GitHub environment variables that start with
BICEP_
to be used as shell environment variables in the Bicep deployment (as variables configured in GitHub environments for deployment are not exposed to the shell by default). - Bicep deployment: Uses the
azure/arm-deploy@v1
action to deploy the infrastructure defined in the Bicep templates. - Convert Bicep outputs to infra-outputs.json: Converts the outputs from the Bicep deployment to a
infra-outputs.json
file. - Upload infra-outputs.json: Uploads the
infra-outputs.json
file as an workflow artifact, making it accessible to other reusable workflows in the CD pipeline. - JSON to variables: reads the
infra-outputs.json
file and exposes the outputs as shell environment variables (prefixed by$INFRA_
), making all the Bicep outputs accessible to downstream job steps.
3. Utilize Bicep Outputs in Post-Infrastructure Deployment Steps
Building upon the previous point, you can now use the Bicep outputs stored as JSON in subsequent deployment steps. This approach simplifies post-infrastructure deployments by providing easy access to resource details like names, connection strings, and endpoints.
Example: Deploying an Application Using Bicep Outputs
Below is an example of how to use the Bicep outputs in a deployment workflow to deploy an application to Azure App Service.
Reference application deployment workflow (deploy_app.yaml
):
name: Deploy Application to Azure App Service
on:
workflow_call: # allow to be called from other workflows
inputs:
environment_name:
required: true
type: string
workflow_dispatch: # allow triggering manually
inputs:
environment_name:
description: "GitHub deployment environment to pull variables and secrets from"
required: true
type: environment
# if triggering manually, users provide the run ID of a previous deployment run to grab infra outputs from
deployment_run_id:
description: "Run ID of the deployment to use infrastructure variables from"
required: true
type: number
jobs:
deploy_app:
runs-on: ubuntu-latest
environment: ${{ inputs.environment_name }}
steps:
# Download the Bicep outputs artifact
- name: Download infra-outputs.json
uses: actions/download-artifact@v4
with:
name: infra-outputs-json
run-id: ${{ inputs.deployment_run_id || github.run_id }}
# Expose bicep outputs as shell variables with a $INFRA_ prefix
- name: JSON to variables
shell: bash
run: |
cat infra-outputs.json | jq -r '[paths(scalars) as $path | {"key": $path | map(if type == "number" then tostring else . end) | join("_"), "value": getpath($path)}] | map("\(.key)=\(.value)") | .[]' | while read line;do
echo "${{ inputs.prefix }}${line}" >> $GITHUB_ENV
done
- name: Azure CLI Login
uses: azure/login@v2
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
# Use the $INFRA_* environment variables to get resource names output by Bicep
- name: Deploy Application to App Service
run: |
APP_SERVICE_NAME="$INFRA_names_appService"
RESOURCE_GROUP="$INFRA_resourceGroup"
echo "Deploying to App Service: $APP_SERVICE_NAME in Resource Group: $RESOURCE_GROUP"
# Deploy application code to the App Service
az webapp deploy \
--name "$APP_SERVICE_NAME" \
--resource-group "$RESOURCE_GROUP" \
--src-path ./app \
--type zip
Explanation of Key Steps
- Download infra-outputs.json: Downloads the infra-outputs.json artifact containing the Bicep outputs from the infrastructure deployment.
- JSON to Variables: Converts the JSON file into environment variables prefixed with
$INFRA_
for easy access. - Azure CLI Login: Authenticates with Azure using provided credentials.
- Deploy Application to App Service:
- Retrieves the App Service name and resource group from the environment variables set from Bicep outputs.
It assumes the Bicep has an output
names.appService
, which becomes available as$INFRA_names_appService
in the CI shell. - Uses the Azure CLI to deploy the application code to the specified App Service.
- Retrieves the App Service name and resource group from the environment variables set from Bicep outputs.
It assumes the Bicep has an output
Conclusion
Modularizing pipelines with reusable workflows and converting Bicep outputs to JSON enables you to create developer-friendly CI/CD pipelines on Azure. Utilizing these outputs in subsequent deployment steps not only simplifies the process but also enhances the overall efficiency of your pipeline.
Implementing these strategies enables your development team to:
- Focus on Code Quality: Spend more time improving the application rather than managing complex pipelines.
- Increase Productivity: Reduce the overhead associated with pipeline maintenance.
- Promote Consistency: Ensure consistent deployment processes across different environments and projects.
Start adopting these practices today to streamline your CI/CD pipelines and accelerate your development workflow.
*The featured image was generated using ChatGPT’s DALL-E tool.