Announcing General Availability of YAML CD features in Azure Pipelines
Azure Pipelines YAML CD features now generally available
We’re excited to announce the general availability of the Azure Pipelines YAML CD features. We now offer a unified YAML experience to configure each of your pipelines to do CI, CD, or CI and CD together.
Releases vs. YAML pipelines
Azure Pipelines supports continuous integration (CI) and continuous delivery (CD) to test, build and ship your code to any target – repeatedly and consistently. You accomplish this by defining a pipeline. CD pipelines can be authored using the YAML syntax or through the visual user interface (Releases).
You can create and configure release pipelines in the web portal with the visual user interface editor (Releases). A release pipeline can consume and deploy the artifacts produced by the CI pipelines to your deployment targets. On the other hand, you can define the CI/CD workflows in a YAML file –
azure-pipelines.yml. The pipelines file is versioned with your code. It follows the same branching structure. As result, you get validation of the changes through code reviews in pull requests and branch build policies. Most importantly, the changes are in the version control with the rest of the codebase so you can easily identify the issue.
YAML CD features introduces several new features that are available for all organizations using multi-stage YAML pipelines. Some of the highlights include.
Multi-stage YAML pipelines (for CI and CD)
Stages are the major divisions in a pipeline: “build app”, “Run tests”, and “deploy to Prod” are good examples of stages. They are a logical boundary in your pipeline at which you can pause the pipeline and perform various checks. Stages may be arranged into a dependency graph. For example, “run dev stage before the QA”.
Resources in YAML pipelines
A resource defines the type of artifact used by a pipeline. Resources provide full traceability of the artifacts consumed in your pipeline. It includes artifact, version, associated commits, and work-items. Most importantly, fully automate the DevOps workflow by subscribing to trigger events on your resources. Supported type of resources are: pipelines, builds, repositories, containers, and packages.
resources: pipelines: [ pipeline ] builds: [ build ] repositories: [ repository ] containers: [ container ] packages: [ package ]
Environments and deployment strategies
An environment is a collection of resources that can be targeted for deployment from a pipeline. Environments can include Kubernetes clusters, Azure web apps, virtual machines, databases etc. Typical examples of environment names are Dev, Test, QA, Staging, and Production.
Kubernetes and Virtual Machine resources in environment
Kubernetes resource view provides the status of objects within the mapped Kubernetes namespace. It overlays pipeline traceability on top so you can trace back from the Kubernetes object to the pipeline and to the commit. Virtual machine resource view lets you add your VMs that are on any cloud or on-premises. As a result, rollout application updates and track deployments to the virtual machines.
Approvals and checks on resources
Pipelines rely on resources such as environments, service connections, agent pools, and library items. Checks enable the resource owner to control whether a stage in a pipeline can consume the resource. As an owner, you can define the checks that must be met prior a stage consuming that resource can start. For example, a manual approval checks on an environment ensures that deployment can happen only after the reviewers have signed-off.
Review apps for collaboration
Review apps works by deploying every pull request from your git repository to a dynamically created resource in an environment. The team can see the changes in the PR, as well as how they work with other dependent services before they’re merged into the main branch. As a result, you to shift-left the code quality and improve productivity.
Refreshed UX for service connections
We have improved the service connection user experience. Further, enabled configuring approvals and checks on service connections. For approvals, we follow segregation of roles between the resource owners and developers. For instance, any pipeline runs that use a connection with the checks enabled will pause the workflow.
Lastly, we want to thank all our users who have adopted Azure Pipelines. If you’re new to Azure Pipelines, get started for free on our website.. Learn what’s new and what you can do with Azure Pipelines. As always, we’d love to hear your feedback and comments. Feel free to comment on this post or tweet at @AzureDevOps with your thoughts.
Nice that one for GA Multi-stage YAML pipelines (for CI and CD), any plans for allowing to chose version of resources pipelines when scheduling Release in Yaml or option to have possibility to use variables to select version of resources, missing also option to have environment deployment on higher level then job ?
Resource version picker is coming soon. It is getting rolled out. Stage is the boundary that is currently supported. Deployment at environment level is not planned for anytime soon
Nice to hear its now GA.
does this support using of Deployment Groups? without it we can’t move to this
You might want to take a look to the ‘VM Environment’ resources which seems to act similar to the ‘Deployment Groups’
(you can even install that agent into physical servers, so not sure why do they name it like that :P)
You should use environments, unless you require automated creation. There is no Azure VM extension or unattended agent provisioning of VM environments yet.
Everything can be automated using Terraform. In fact Terraform is provided in Azure Cloud Shell.
Take a look at these docs:. https://docs.microsoft.com/en-us/azure/developer/terraform/install-configure
We are running Terraform and Packer in our Pipelines to build and spin up windows images.
That’s correct. VM environment is the replacement for Deployment Groups in YAML pipelines. Register VM script generated is like that of deployment group except the target. To clarify, VM registration in environment targets the Environment. You can use the same script to automate.
I wish we could select an existing agent to bind instead of having to re-run the script.
Good to see all the improvements and see it in GA, however until it supports Deployment Queue Settings (Similar to legacy Releases) we cannot move to this.
Support for concurrency control is going to be available soon as “checks” on the environment. Stay tuned
The classic CD has great REST APIs – are public REST APIs planned for YAML CD?
We’ve been using YAML pipelines for a few months now and our observation is that they are nearly identical.
The classic pipeline has a yaml view that you can use to convert to a yaml pipeline. So indeed, REST API’s are supported. Here’s the docs: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/http-rest-api?view=azure-devops
REST APIs for YAML CD constructs is on the backlog. No timelines yet, but will be taken for sure.
what about Variables by each environment? in the CD pipelines?
Indeed! I think they want you to store these variables now as actual code in the YAML file.
You can use the Library to create groups of variables, and then load those at different points in the YAML – the pipeline as a whole has a variables section that you can load a library set into, and then each stage can also load in variable groups scoped as you would expect.
The editing experience isn’t anywhere as good as the scoped variables on a release pipeline, with lots of copy and paste and chances for errors though 🙁
There are actually three places where variables may be stored now:
1. Directly in the Pipeline yaml code.
2. In a Variables sections when you edit the pipeline in your browser. This is actually a neat feature that allows you to override the variables with the override option set, per manual run.
3. The Library. You will want to use this if you have secret keys, or if you want to link with the Azure Vault.
Using a combination of all 3 allows a lot of flexibility.
When will Azure DevOps Server get this? Thanks!
According to the roadmap item linked to from the Azure DevOps Feature TimelineAzure DevOps Feature TimelineAzure DevOps Feature Timeline this should be added in Azure DevOps Server 2020.
This does of cause not state precisely ‘when’ but at least give as idea.
We are looking forward to this too :-).
I assume this still does not support the 1 feature that everybody that uses “classic” releases is still waiting for – the ability to deploy to different environments using a single pipeline but not automated. Note that this is not the same as a manual approval and never has been. The team seems to keep lumping them together. The YAML pipelines require that all the stages either get run or opt out. In other words we build, deploy to Dev, maybe run tests, deploy to QA, maybe have a manual approval and then deploy to Prod. But the pipeline is “in progress” until all this completes. For companies that CD to Dev every day but only send a handful to QA maybe once a sprint and to Prod every couple of sprints you end up with either failed pipelines or multiple active ones.
Is the highly requested feature of being able to use YAML pipelines and later (days, weeks even months) deploy to other stages using the same release like we can in classic or are we stuck using classic until MS realizes we don’t deploy the way you do?
I think you’re looking for this developercommunity idea – I don’t think it’s done yet, but at least it is on the roadmap. 🙂
This can be accomplished using multiple pipelines. Why would you want pipelines stacking up in a partially finished state?
Pipelines can feed to other pipelines. Have another pipeline for approvals.
I totally agree with Michael. This is the single feature that is keeping us from moving towards YAML for deployments. Sure, it can be accomplished via approvals, but that means constantly getting bombarded with emails everyday for every PR that gets merged in, when really we only want to push one version to Prod each sprint. Using approvals is a very clunky work around. I just want every PR merged in to get pushed to Dev and Int automatically, but want to manually push a button to go to RC and Prod, without having to tell everyone in the company to setup special email rules to send all approval emails to their trash, effectively making the approval notifications worthless.
@ConsulTent Support, this is in no way solved by multiple pipelines. In fact that would make it worse. With classic release I can go to any of my 80+ pipelines that I manage and see at a glance that release 1 of Prod A is currently in QA whereas release 2 of Prod A is in Dev. I cannot remotely imagine how that would look if I had a separate pipeline for each of my stages. Release pipelines are nothing more than glorified back of artifacts with deployment instructions. There is no reason why you would ever treat a deployment into an environment of the same artifacts as a brand new release, at least in companies I’ve been at.
The other thing is that the pipeline is partially finished. A release isn’t official until it gets deployed to Prod. There can be any # of releases to Dev and likely less releases to QA that ultimately cumulate into a Prod release. At any point in time a QA release could be promoted to Prod. Meanwhile you could have multiple other releases in other stages. The release isn’t finished but the stage is. Multiple pipelines doesn’t solve the underlying issue is just replicates how many pipelines you’re dealing with.
The ask from the community has always been to have the same ability as classic release in terms of deploying to stages on demand. Until YAML releases can do this there is no reason for anyone to complicate their existing, working releases by any hacky approach. If multiple pipelines work for you then great but it is not the solution to the problem.
@Michael: Thank you for posting this – it has saved me a heap of time of figuring this out myself!
@Roopesh (author): it would be awesome if you included a link in the post to the roadmap/future plans, especially around feature parity. I’m really looking forward to when I can finally get my release definitions into git.
Without the ability to limit concurrency/parallelism of a stage, YAML releases are still no use to me, sorry.
Have you looked into the dependencies feature? Allows for control of concurrent/series running, fan in/out, etc.
You’re looking for the maxParallel https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema#job
Concurrency control on the resources like Environments, Service Connections etc. is coming soon. Please stay tuned
Still struggling to find documenting on how deployments get linked to Kubernetes resources?
Only seems to cover kubectl. What about helm deployments?
There is built in Kubectl deploy steps that support Helm charts as well. Please check it out
Is there planned work to map environments to stages for work item tracking like we get with classic so that we can see at a glance where a bug/story/task is deployed to from within the card
Environments has a view of work items and commits that were deployed to it. Does that work for you?