Containerized CI with Concourse on Azure
What’s Concourse?
Concourse CI is a unique CI application where the concept of Pipelines is first class throughout the entire process. Concourse allows you to create more complex build systems through a series of input and output resources with each step defined by a task manifest and running in its own container.
The real advantages to Concourse are the out of the box support of containerized pipeline steps which prevents any kind of configuration pollution between steps as well as its ability to deploy highly scalable solutions suitable for large application development.
The Problem
You may have heard of the recent announcement of Cloud Foundry Support on Microsoft Azure. Cloud Foundry is an open source PaaS that is deployed by a deployment management tool called BOSH. BOSH works by using Cloud Platform Integration (CPI) packages to handle the IaaS details of each cloud provider. In order to have Cloud Foundry on Azure, you need to be able to support BOSH deployments on Azure which requires an Azure CPI package.
Since Cloud Foundry is used by some of the largest enterprises in the world, it’s essential to support a Continuous Integration process for the CPI package. This ensures further code changes to the CPI won’t regress future Cloud Foundry deployments on Azure.
The kind of testing needed for the CPI package needs to be quite exhaustive and must be end to end. In order to do this, we need to support the following Continuous Integration Steps:
- Build the CPI Package
- Deploy the Initial Azure Infrastructure for BOSH
- Deploy the BOSH Director with the Azure CPI Package
- Run the BOSH Acceptance Test Suite Against the BOSH Director
- Publish the Passed CPI Build to a git Branch
We partnered with Pivotal, the maintainers of Cloud Foundry, to create a CI pipeline that can test the CPI’s functionalities with high confidence. This enables the open source Cloud Foundry community with continuous updates for Cloud Foundry on Azure as well as support for enterprises who are using Cloud Foundry on Azure to deploy applications and services.
The Solution
Because the build process required is quite complex, with very distinct parts, we decided to use Concourse CI to automate the build with a series of well-defined, containerized, and isolated steps. This isolation means that system changes made in one container won’t “pollute” settings in another container allowing for a clean and well-known state at each build step.
Concourse Resources
Between each step, the state of the build is stored off into Concourse Resources, which can be anything from a git repository, a semantic version number, to an S3 Bucket. Resources are used as inputs and outputs between each step allowing each concourse Task to “connect” to one another by taking the output of one task and feeding it as the input to another. Concourse Resources implement an interface specification which allows for it to be used by Concourse CI.
A Concourse deployment can run locally using Vagrant and can be deployed to any infrastructure that supports BOSH. In the spirit of dog-fooding, we deployed Concourse to Azure using the very same Azure CPI package we were testing. This allowed us to actually use the CPI as well as test it in an automated fashion.
Concourse does a great job at visualizing your build pipeline using an easy-to-understand block diagram with resources and build tasks connected via input and output lines. This easily allows you to see the real-time status of your build and to re-run any discrete step.
You can see the Bosh Azure CPI Release CI Pipeline below. It is composed of 6 discrete and isolated tasks.
Each black block with lines coming in and out is a Concourse Resource, and each green block is a Concourse Task. The entire pipeline is described by a YAML file, which describes each task as well as its input and output resources. If you’re curious about the details of this pipeline as defined by its YAML, you can checkout the actual CI pipeline YAML file here.
Build Tasks
If you look closer at our CI pipeline definition, you will notice our CI tasks refer to a Docker image.
Container
As we mentioned earlier, each build step is executed in its own container, specified by an image hosted in a Docker registry. Concourse actually runs containers with garden-linux, which is compatible with Open Container Specification images. For all intents and purposes, this is an implementation detail and your container will run exactly as you would expect it to run if you ran it with Docker.
To understand details about that container, take a look at our image repository.
Build task definition
Build tasks are defined by a simple YAML structure similar to this:
---
platform: linux
image: docker:///sedouard/azure-cpi-release
**Inputs:**
- name: bosh-cpi-release
- name: version-semver
run:
path: bosh-cpi-release/ci/tasks/build-candidate.sh
This definition contains several key components for this build task:
- Reference to the Docker image to use for this step
- Inputs to use that are described in the main pipeline’s YAML file
- Reference to the script file to execute in this step
Below are quick overviews of each build step, what it does, and what it produces. A link is provided to the actual definition in code for each step.
Build-Candidate
Code available on GitHub
Inputs:
- version-semver – The semantic version of this build
- bosh-cpi-release-in – The git repository of the CPI to create the BOSH release from
What it does:
Builds the BOSH release candidate using a series of standard steps specific to creating a BOSH release.
Outputs:
- bosh-cpi-dev-artifacts – The current BOSH release tarball
- version-semver – The same semantic version of the build passed as an input
Azure-Provision
Code available on GitHub
Inputs:
- version-semver – The semantic version of this build
- bosh-cpi-release-in – The git repository of the CPI to create the BOSH release from
What it does:
Provisions the basic azure infrastructure required to deploy BOSH-managed infrastructure on Azure. This mainly includes a VNet, two subnets, a resource group, a storage account, and two public IP addresses. All the gritty details of how we extract the deployment information from Azure using the CLI can be found in this task’s shell script.
Outputs:
- azure-exports – A generated script file that includes the public IP addresses to use for Cloud Foundry and the BOSH director. This also includes specific details on connecting to the created Azure Storage account
Deploy-Director
Code available on GitHub
Inputs:
- bosh-cpi-dev-artifacts – The current BOSH release tarball
- version-semver – The semantic version of this build
- azure-exports – A generated script file that includes the public IP addresses to use for Cloud Foundry and the BOSH director
- bosh-init – The bosh-init tarball for Linux
- bosh-release – The BOSH release tarball to use for the director
- bosh-cpi-release-in – The git repository of the CPI to create the BOSH release from
- ubuntu-director-state-file – A file that records the current deployment state of the director – to enable re-runs by deleting and re-provisoning the director on the same azure resource group deployment
What it does:
Uses the bosh-init CLI to deploy a rendered manifest file template that describes the virtual machines and networking infrastructure to have a BOSH director capable of deploying BOSH releases.
The deploy-director step uses the azure-exports input file created as a result of the completion of the azure-provision step. With the azure-exports input file, the public IP address and storage account keys provisioned in the previous step is automatically injected. This information will be used to provision the BOSH director.
Outputs:
- ubuntu-director-state-file – The file that represents the current deployment state of the director. Used by any future re-runs of the
Deploy-Directorstep. - bosh-cpi-dev-artifacts – The current BOSH release tarball – passes this along from the input
- bosh-cpi-release-in – The git repository of the CPI to create the BOSH release from – passes this along from inputs
Run-Bats
Code available on GitHub
Inputs:
- azure-exports – A generated script file that includes the public IP addresses to use for Cloud Foundry and the BOSH director
- azure-ubuntu-stemcell – The base image to use for VMs of any BOSH deployment on Ubuntu. This includes the VM agent for Azure.
- bosh – The official BOSH Repo – used to run BATs
- bosh-cpi-dev-artifacts – The current BOSH release tarball for this build
- bosh-cpi-release-in – The git repository of the CPI to create the BOSH release from
What it does:
Runs the standard set of BOSH Acceptance Tests (BATs) against the BOSH director. This runs a series of steps such as bosh deployment, bosh deploy, verification of the state of the BOSH director, and verification of deployments after each command. This task is triggered by the successful completion of the Deploy-Director task.
Outputs:
- bosh-cpi-dev-artifacts – The current BOSH release tarball – passed along from input
- bosh-cpi-release-in – The git repository of the CPI to create the BOSH release from – passed along from input
Promote Candidate
Code available on GitHub
Inputs:
- bosh-cpi-dev-artifacts – The current BOSH release tarball – passes this along from the input
- bosh-cpi-release-in – The git repository of the CPI to create the BOSH release from – passes this along from inputs
What it does:
Promote Candidate will be triggered by a successful completion of the Run-Bats task. It will take the current target git branch within bosh-cpi-release-in, deploy the promoted build of the CPI to S3, and publish the current state of the branch to a new release branch. You can check out the detailed steps in the shell script.
Outputs:
- bosh-cpi-release-out – The promoted BOSH release package
A Complete End to End CI System
The above steps comprise the full end-to-end pipeline steps that are required to validate the Azure CPI similarly to how an end user would use it.
Pivotal will continue to use this pipeline to support the Azure CPI and BOSH deployments on Azure. If you’re wondering how you can use BOSH to deploy Cloud Foundry on Azure, you should head over to our post about that topic.
Getting Started with Concourse CI
The great thing about this collaboration is that you too can create a scalable, CI system that’s completely open source. We have collaborated with the Concourse CI maintainers to create the Azure Documentation for Concourse CI.
These instructions will walk you through creating your own infrastructure, BOSH director, and deploying Concourse to Azure IaaS. You’ll be able to scale this deployment with as many VMs as you need for your CI workload needs.
Light
Dark
0 comments