In our recent engagement, our team crafted reusable and extensible pipelines designed to enhance consistency, improve readability, and drive developer productivity.\nIn this blog, we share our insights to help you elevate your own pipeline efficiency.\nContinuous Integration and Continuous Deployment (CI\/CD) pipelines are critical for ensuring efficient and reliable code deployment.\nAs projects grow, these pipelines can become increasingly complex, making them difficult to manage and extend.<\/p>\n
By adopting the following three approaches, you can streamline pipeline management, lower operational overhead, and boost productivity:<\/p>\n
Let’s dive into each of these approaches with detailed explanations and code examples.<\/p>\n
GitHub Actions supports reusable workflows, allowing one workflow to call another.\nBy modularizing your CI\/CD pipelines using this pattern, you enhance maintainability and readability across your workflows.<\/p>\n
Here’s how to create a modularized CI pipeline using GitHub Actions to validate code on pull requests or merges to protected branches.<\/p>\n
Create a Each job references a separate re-usable workflow:<\/p>\n Reference linting workflow The Similarly, create a modularized CD pipeline to deploy the application.\nThis workflow is divided into several jobs, each responsible for a specific deployment task such as parsing environment information, deploying infrastructure, or pushing Docker images.<\/p>\n This modular approach ensures that each deployment step in the CD pipeline is self-contained and can be reused or triggered manually for a partial deployment if required.<\/p>\n Reference Each job in the CD pipeline focuses on a specific task:<\/p>\n When deploying Azure infrastructure with Bicep, you often need to pass outputs (e.g. resource names, endpoint URIs, etc) to subsequent steps or workflows to configure the newly deployed infrastructure.<\/p>\n By capturing Bicep outputs and storing them as JSON artifacts, you can easily pass infrastructure details to other jobs or workflows. This method will streamline your CD pipeline.<\/p>\n Below is an easy-to-understand code example demonstrating how to deploy infrastructure using Bicep and capture its outputs for better integration.<\/p>\n Contents of ci.yaml<\/code> that calls other reusable workflows for its linting and test tasks:<\/p>\nname: Continuous Integration\non:\n workflow_call: # allow to be called from other workflows\n workflow_dispatch: # allow triggering manually\n pull_request: # on any PR\njobs:\n linting:\n uses: .\/.github\/workflows\/linting.yaml\n test_dotnet:\n uses: .\/.github\/workflows\/test_dotnet.yaml\n test_python:\n uses: .\/.github\/workflows\/test_python.yaml<\/code><\/pre>\n\n
linting.yaml<\/code>: Contains steps for code linting.<\/li>\ntest_dotnet.yaml<\/code>: Contains steps for .NET unit tests.<\/li>\ntest_python.yaml<\/code>: Contains steps for Python unit tests.<\/li>\n<\/ul>\nlinting.yaml<\/code>:<\/p>\nname: Linting\non:\n workflow_call: # allow to be called from other workflows\njobs:\n lint:\n runs-on: ubuntu-latest\n steps:\n - uses: actions\/checkout@v3\n - name: Run Linting\n run: |\n # Add linting commands here\n echo \"Linting code...\"<\/code><\/pre>\ntest_{language}.yaml<\/code> workflows can be established in a similar way to call setup the language-specific toolchains and call dotnet test<\/code> or pytest<\/code>.<\/p>\nb. Modularized Continuous Deployment (CD) Pipeline<\/h3>\n
cd.yaml<\/code>:<\/p>\nname: Continuous Deployment\n\non:\n workflow_dispatch: # allow triggering manually\n inputs:\n # manually triggered runs have GITHUB_REF_NAME as a branch, not a tag so we need to manually ask for the environment\n environment_name:\n description: \"GitHub deployment environment to pull variables and secrets from\"\n required: true\n type: environment\n push:\n branches:\n # pushing to a branch called releases\/foo will automatically trigger a deployment using the 'foo' environment (as configured in GitHub Environments)\n - releases\/**\n\njobs:\n # Run CI before deployment\n ci:\n uses: .\/.github\/workflows\/ci.yaml\n\n detect_env:\n needs: [ci] # don't proceed with environment deployment unless CI succeeded\n runs-on: ubuntu-latest\n outputs:\n # environment name is the name of the GitHub Deployment Environment to reference\n environment_name: ${{ steps.parse_ref.outputs.environment_name }}\n # build_version will be used to tag docker images\/packages\/artifacts\n build_version: ${{ steps.parse_ref.outputs.build_version }}\n steps:\n # Obtain environment name from Git tag\/branch when possible\n - id: parse_ref\n name: Parse Environment from Git Reference\n run: |\n build_version=\"${{ github.sha }}\"\n if [ ! -z \"${{ inputs.environment_name }}\" ];then\n # user specified environment; do not auto-detect\n environment_name=\"${{ inputs.environment_name }}\"\n echo \"Manually specified environment '$environment_name', setting build version to SHA '$build_version'\"\n else\n # no environment specified; use the current branch\/tag name without the 'releases\/' prefix\n environment_name=\"${{ github.ref_name }}\"\n environment_name=\"${environment_name##releases\/}\"\n fi\n\n echo \"Using environment '$environment_name' with build version '$build_version' from ref ${{ github.ref_name }}\"\n echo \"environment_name=$environment_name\" >> $GITHUB_OUTPUT\n echo \"build_version=$build_version\" >> $GITHUB_OUTPUT\n\n deploy_infra:\n needs: [detect_env]\n uses: .\/.github\/workflows\/deploy_infra.yaml\n secrets: inherit\n with:\n environment_name: ${{ needs.detect_env.outputs.environment_name }}\n build_version: ${{ needs.detect_env.outputs.build_version }}\n\n docker_build_and_push:\n permissions:\n contents: read\n packages: write\n needs: [deploy_infra]\n uses: .\/.github\/workflows\/docker_build_and_push.yaml\n secrets: inherit\n with:\n environment_name: ${{ needs.detect_env.outputs.environment_name }}\n build_version: ${{ needs.detect_env.outputs.build_version }}\n\n deploy_app:\n needs: [docker_build_and_push]\n uses: .\/.github\/workflows\/deploy_app.yaml\n secrets: inherit\n with:\n environment_name: ${{ needs.detect_env.outputs.environment_name }}<\/code><\/pre>\n\n
ci<\/code>: Runs the CI workflow to ensure code quality before deployment.<\/li>\ndetect_env<\/code>: Determines the deployment environment, either use-specified or from the Git branch\/tag name.<\/li>\ndeploy_infra<\/code>: Deploys infrastructure using Bicep templates.<\/li>\ndocker_build_and_push<\/code>: Builds and pushes Docker images.<\/li>\ndeploy_app<\/code>: Deploys the application to the target environment.<\/li>\n<\/ul>\n2. Convert Bicep Outputs to JSON for Better Integration<\/h2>\n
Reference infrastructure deployment workflow<\/h3>\n
deploy_infra.yaml<\/code>:<\/p>\nname: Deploy infrastructure\n\non:\n workflow_call: # allow to be called from other workflows\n inputs:\n environment_name:\n required: true\n type: string\n build_version:\n required: true\n type: string\n workflow_dispatch: # allow triggering manually\n inputs:\n environment_name:\n description: \"GitHub deployment environment to pull variables and secrets from\"\n required: true\n type: environment\n build_version:\n description: \"Build version (usually commit SHA) to tag images with\"\n required: true\n type: string\n\njobs:\n iac:\n runs-on: ubuntu-latest\n # pull SUBSCRIPTION_ID, RESOURCE_GROUP variables and AZURE_CREDENTIALS json secret in the GitHub Environment\n # see https:\/\/docs.github.com\/en\/actions\/writing-workflows\/choosing-what-your-workflow-does\/using-environments-for-deployment\n environment: ${{ inputs.environment_name }}\n steps:\n - name: Checkout code\n uses: actions\/checkout@v4\n\n - name: Login to Azure\n uses: azure\/login@v2\n with:\n creds: ${{ secrets.AZURE_CREDENTIALS }}\n\n # Map in all variables that start with BICEP_ from GitHub deployment environment as environment variables\n - name: Set env vars from vars context JSON\n id: ghenv-to-envvar\n env:\n # https:\/\/github.com\/actions\/runner\/issues\/1656#issuecomment-1030077729\n VARS_JSON: ${{ toJSON(vars) }}\n run: |\n echo $VARS_JSON | jq -r 'to_entries | .[] | select(.key | startswith(\"BICEP_\")) | \"\\(.key)=\\(.value)\"' >> $GITHUB_ENV\n\n - id: bicep-deploy\n name: \"Bicep deployment\"\n uses: azure\/arm-deploy@v1\n env:\n # Map in any other variables that aren't directly set in GitHub environment\n BICEP_ENVIRONMENT_NAME: ${{ inputs.environment_name }}\n BICEP_RESOURCE_GROUP: ${{ vars.RESOURCE_GROUP }}\n with:\n subscriptionId: ${{ vars.SUBSCRIPTION_ID }}\n resourceGroupName: ${{ vars.RESOURCE_GROUP }}\n template: deployment\/iac\/main.bicep\n parameters: deployment\/iac\/main.bicepparam\n\n - name: Convert Bicep outputs to infra-outputs.json\n env:\n ENV_VARS_JSON: ${{ toJSON(steps.bicep-deploy.outputs) }}\n run: |\n # non-string outputs are stringified by GitHub Actions - parse the values as JSON when possible\n jq -re 'to_entries | map({(.key): (.value | fromjson? \/\/ .)}) | add' <<< \"$ENV_VARS_JSON\" > infra-outputs.json\n\n - name: Upload infra-outputs.json\n uses: actions\/upload-artifact@v4\n with:\n name: infra-outputs-json\n path: infra-outputs.json\n if-no-files-found: error\n overwrite: true\n\n # Make Bicep outputs available as prefixed env vars\n # In other workflows, remember to download the artifact above first\n - name: JSON to variables\n shell: bash\n run: |\n cat infra-outputs.json | jq -r '[paths(scalars) as $path | {\"key\": $path | map(if type == \"number\" then tostring else . end) | join(\"_\"), \"value\": getpath($path)}] | map(\"\\(.key)=\\(.value)\") | .[]' | while read line;do\n echo \"${{ inputs.prefix }}${line}\" >> $GITHUB_ENV\n done\n\n # now, a Bicep output 'foo.bar' can be referenced as '$INFRA_foo_bar' in subsequent steps\n # ...<\/code><\/pre>\nExplanation of Key Steps<\/h3>\n