Using containerized services in your pipeline

Matt Cooper

Matt

Azure Pipelines has supported container jobs for a while now. You craft a container with exactly the versions of exactly the tools you need, and we’ll run your pipeline steps inside that container. Recently we expanded our container support to include service containers: additional, helper containers accessible to your pipeline.

Service containers let you define the set of services you need available, containerize them, and then have them automatically available while your pipeline is running. Azure Pipelines manages the starting up and tearing down of the containers, so you don’t have to think about cleaning them up or resetting their state. In many cases, you don’t even have to build and maintain the containers – Docker Hub has many popular options ready to go. You simply point your app at the services, typically by name, and everything else is taken care of.

So how does it work? You tell Azure Pipelines what containers to pull, what to call them, and what ports & volumes to map. The pipelines agent manages everything else. Let’s look at two examples showing how to use the feature.

Basic use of service containers

First, suppose you need memory cache and proxy servers for your integration tests. How do you make sure those servers get reset to a clean state each time you build your app? With service containers, of course:

When the pipeline runs, Azure Pipelines pulls three containers: Ubuntu 16.04 to run the build tasks in, nginx for a proxy server, and Redis for a cache server. The agent spins up all three containers and networks them together. Since everything is running on the same container network, you can access the services by hostname: that’s what the curl nginx and redis-cli -h redis ping  lines are doing. Of course, in your app, you’d do more than just ping the service – you’d configure your app to use the services. When the job is complete, all three containers will be spun down.

Combining service containers with a matrix of jobs

Suppose you’re building an app that supports multiple different database backends. How do you easily test against each database, without maintaining a bunch of infrastructure or installing multiple server runtimes? You can use a matrix with service containers, like this:

In this case, the listed steps will be duplicated into two jobs, one against Postgres 10 and the other against Postgres 11.

Service containers work with non-container jobs, where tasks are running directly on the host. They also support advanced scenarios such as defining your own port and volume mappings; see the documentation for more details. Like container jobs, service containers are available in YAML-based pipelines. [Edit, 2019-01-24: this feature has not finished rolling out everywhere on Azure DevOps. If it doesn’t work for you, wait a few days and give it another try. Sorry for the confusion!]

Matt Cooper
Matt Cooper

Senior Program Manager, Azure Pipelines

Follow Matt   

1 Comments
Jaap-Willem van den Berg
Jaap-Willem van den Berg 2019-04-23 05:40:12
Hi, I'm trying to use the service containers in combination with Azure Container Registry. I have created a 'Service Connection' for 'Docker Registry' with the name 'myacr'. I have set Authorization for the build script to use this Service Connection. I keep getting the error: ##[error]Value cannot be null.Parameter name: registryServer ##[debug]System.ArgumentNullException: Value cannot be null.  Am I doing something wrong or is this a known error? My yaml: resources:  repositories:   - repository: self containers:  - container: test    image: mysomething.azurecr.io/myimage    endpoint: myacr pool:  vmImage: 'ubuntu-16.04' services:  test: test