The real advantages to Concourse are the out of the box support of containerized pipeline steps which prevents any kind of configuration pollution between steps as well as its ability to deploy highly scalable solutions suitable for large application development.<\/p>\n
The Problem<\/h3>\n
You may have heard of the recent announcement of Cloud Foundry Support on Microsoft Azure<\/a>. Cloud Foundry<\/a> is an open source PaaS that is deployed by a deployment management tool called BOSH<\/a>. BOSH works by using Cloud Platform Integration (CPI) packages to handle the IaaS details of each cloud provider. In order to have Cloud Foundry on Azure, you need to be able to support BOSH deployments on Azure which requires an Azure CPI package.<\/p>\n Since Cloud Foundry is used by some of the largest enterprises in the world, it\u2019s essential to support a Continuous Integration process for the CPI package. This ensures further code changes to the CPI won\u2019t regress future Cloud Foundry deployments on Azure.<\/p>\n The kind of testing needed for the CPI package needs to be quite exhaustive and must be end to end. In order to do this, we need to support the following Continuous Integration Steps:<\/p>\n <\/p>\n <\/p>\n We partnered with Pivotal, the maintainers of Cloud Foundry, to create a CI pipeline that can test the CPI\u2019s functionalities with high confidence. This enables the open source Cloud Foundry community with continuous updates for Cloud Foundry on Azure as well as support for enterprises who are using Cloud Foundry on Azure to deploy applications and services.<\/p>\n Because the build process required is quite complex, with very distinct parts, we decided to use Concourse CI<\/a> to automate the build with a series of well-defined, containerized, and isolated steps. This isolation means that system changes made in one container won\u2019t \u201cpollute\u201d settings in another container allowing for a clean and well-known state at each build step.<\/p>\n Between each step, the state of the build is stored off into Concourse Resources<\/a>, which can be anything from a git repository<\/a>, a semantic version number<\/a>, to an S3 Bucket<\/a>. Resources are used as inputs and outputs between each step allowing each concourse A Concourse deployment can run locally using Vagrant and can be deployed to any infrastructure that supports BOSH. In the spirit of dog-fooding, we deployed Concourse to Azure using the very same Azure CPI package we were testing. This allowed us to actually use the CPI as well as test it in an automated fashion.<\/p>\n Concourse does a great job at visualizing your build pipeline using an easy-to-understand block diagram with resources and build tasks connected via input and output lines. This easily allows you to see the real-time status of your build and to re-run any discrete step.<\/p>\n You can see the Bosh Azure CPI Release CI Pipeline below. It is composed of 6 discrete and isolated tasks.<\/p>\n Each black block with lines coming in and out is a Concourse Resource, and each green block is a Concourse Task<\/a>. The entire pipeline is described by a YAML file, which describes each task as well as its input and output resources. If you\u2019re curious about the details of this pipeline as defined by its YAML, you can checkout the actual CI pipeline YAML file here<\/a>.<\/p>\n If you look closer at our CI pipeline definition, you will notice our CI tasks refer to a Docker image<\/a>.<\/p>\n As we mentioned earlier, each build step is executed in its own container, specified by an image hosted in a Docker registry. Concourse actually runs containers with garden-linux<\/a>, which is compatible with Open Container Specification<\/a> images. For all intents and purposes, this is an implementation detail and your container will run exactly as you would expect it to run if you ran it with Docker.<\/p>\n To understand details about that container, take a look at our image repository<\/a>.<\/p>\n Build tasks are defined by a simple YAML structure similar to this:<\/p>\n This definition contains several key components for this build task:<\/p>\n Below are quick overviews of each build step, what it does, and what it produces. A link is provided to the actual definition in code for each step.<\/p>\n Code available on GitHub<\/a><\/p>\n Inputs:<\/strong><\/p>\n What it does:<\/strong><\/p>\n Builds the BOSH release candidate using a series of standard steps<\/a> specific to creating a BOSH release.<\/p>\n Outputs:<\/strong><\/p>\n Code available on GitHub<\/a><\/p>\n Inputs:<\/strong><\/p>\n What it does:<\/strong><\/p>\n Provisions the basic azure infrastructure required to deploy BOSH-managed infrastructure on Azure. This mainly includes a VNet, two subnets, a resource group, a storage account, and two public IP addresses. All the gritty details of how we extract the deployment information from Azure using the CLI<\/a> can be found in this task\u2019s shell script<\/a>.<\/p>\n Outputs:<\/strong><\/p>\n Code available on GitHub<\/a><\/p>\n Inputs:<\/strong><\/p>\n What it does:<\/strong><\/p>\n Uses the The Outputs:<\/strong><\/p>\n Code available on GitHub<\/a><\/p>\n Inputs:<\/strong><\/p>\n What it does:<\/strong><\/p>\n Runs the standard set of BOSH Acceptance Tests (BATs) against the BOSH director. This runs a series of steps such as Outputs:<\/strong><\/p>\n Code available on GitHub<\/a><\/p>\n Inputs:<\/strong><\/p>\n What it does:<\/strong><\/p>\n Promote Candidate will be triggered by a successful completion of the Run-Bats<\/a> task. It will take the current target git branch within Outputs:<\/strong><\/p>\n The above steps comprise the full end-to-end pipeline steps that are required to validate the Azure CPI similarly to how an end user would use it.<\/p>\n Pivotal will continue to use this pipeline to support the Azure CPI and BOSH deployments on Azure. If you\u2019re wondering how you can use BOSH to deploy Cloud Foundry on Azure, you should head over to our post about that topic<\/a>.<\/p>\n The great thing about this collaboration is that you too<\/strong> can create a scalable, CI system that\u2019s completely open source. We have collaborated<\/a> with the Concourse CI maintainers to create the Azure Documentation for Concourse CI<\/a>.<\/p>\n These instructions will walk you through creating your own infrastructure, BOSH director, and deploying Concourse to Azure IaaS. You\u2019ll be able to scale this deployment with as many VMs as you need for your CI workload needs.<\/p>\n","protected":false},"excerpt":{"rendered":" Creating a CI pipeline to support the Azure CPI and BOSH deployments on Azure using Concourse CI to automate the build with a series of well-defined, containerized, and isolated steps.<\/p>\n","protected":false},"author":21345,"featured_media":12791,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[16],"tags":[60,118,128],"class_list":["post-2164","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","tag-azure","tag-cloud-foundry","tag-concourse-ci"],"acf":[],"blog_post_summary":" Creating a CI pipeline to support the Azure CPI and BOSH deployments on Azure using Concourse CI to automate the build with a series of well-defined, containerized, and isolated steps.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/2164","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/21345"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=2164"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/2164\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media\/12791"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=2164"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=2164"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=2164"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
The Solution<\/h3>\n
Concourse Resources<\/h4>\n
Task<\/code> to \u201cconnect\u201d to one another by taking the output of one task and feeding it as the input to another. Concourse Resources implement an interface specification which allows for it to be used by Concourse CI.<\/p>\n![\"Animation](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2015-10-17-Containerized-CI-Concourse-Azure-ss1.gif\")
<\/p>\nBuild Tasks<\/h4>\n
Container<\/h5>\n
Build task definition<\/h5>\n
---\r\nplatform: linux\r\nimage: docker:\/\/\/sedouard\/azure-cpi-release\r\n**Inputs:**\r\n - name: bosh-cpi-release\r\n - name: version-semver\r\nrun:\r\n path: bosh-cpi-release\/ci\/tasks\/build-candidate.sh\r\n\r\n<\/code><\/pre>\n\n
Build-Candidate<\/h6>\n
\n
\n
Azure-Provision<\/h6>\n
\n
\n
Deploy-Director<\/h6>\n
\n
bosh-init<\/code> CLI to deploy a rendered manifest file template<\/a> that describes the virtual machines and networking infrastructure to have a BOSH director capable of deploying BOSH releases.<\/p>\ndeploy-director<\/code> step uses the azure-exports<\/code> input file created as a result of the completion of the azure-provision<\/code> step. With the azure-exports<\/code> input file, the public IP address and storage account keys provisioned in the previous step is automatically injected. This information will be used to provision the BOSH director.<\/p>\n\n
Deploy-Director<\/code> step.<\/li>\nRun-Bats<\/h6>\n
\n
bosh deployment<\/code>, bosh deploy<\/code>, verification of the state of the BOSH director, and verification of deployments after each command. This task is triggered by the successful completion of the Deploy-Director<\/a> task.<\/p>\n\n
Promote Candidate<\/h6>\n
\n
bosh-cpi-release-in<\/code>, deploy the promoted build of the CPI to S3, and publish the current state of the branch to a new release branch. You can check out the detailed steps in the shell script<\/a>.<\/p>\n\n
A Complete End to End CI System<\/h2>\n
Getting Started with Concourse CI<\/h2>\n