Announcing General Availability of YAML CD features in Azure Pipelines

Roopesh Nair

Roopesh

Azure Pipelines YAML CD features now generally available

We’re excited to announce the general availability of the Azure Pipelines YAML CD features. We now offer a unified YAML experience to configure each of your pipelines to do CI, CD, or CI and CD together.

Releases vs. YAML pipelines

Azure Pipelines supports continuous integration (CI) and continuous delivery (CD) to test, build and ship your code to any target – repeatedly and consistently. You accomplish this by defining a pipeline. CD pipelines can be authored using the YAML syntax or through the visual user interface (Releases).

Classic vs YAML pipelines

You can create and configure release pipelines in the web portal with the visual user interface editor (Releases). A release pipeline can consume and deploy the artifacts produced by the CI pipelines to your deployment targets. On the other hand, you can define the CI/CD workflows in a YAML file – azure-pipelines.yml. The pipelines file is versioned with your code. It follows the same branching structure. As result, you get validation of the changes through code reviews in pull requests and branch build policies. Most importantly, the changes are in the version control with the rest of the codebase so you can easily identify the issue.

Highlights

YAML CD features introduces several new features that are available for all organizations using multi-stage YAML pipelines. Some of the highlights include.

Multi-stage YAML pipelines (for CI and CD)

Stages are the major divisions in a pipeline: “build app”, “Run tests”, and “deploy to Prod” are good examples of stages. They are a logical boundary in your pipeline at which you can pause the pipeline and perform various checks. Stages may be arranged into a dependency graph. For example, “run dev stage before the QA”.

Multi-stage cd pipelines

Learn more

Resources in YAML pipelines

A resource defines the type of artifact used by a pipeline. Resources provide full traceability of the artifacts consumed in your pipeline. It includes artifact, version, associated commits, and work-items. Most importantly, fully automate the DevOps workflow by subscribing to trigger events on your resources. Supported type of resources are: pipelines, builds, repositories, containers, and packages.

resources:
  pipelines: [ pipeline ]  
  builds: [ build ]
  repositories: [ repository ]
  containers: [ container ]
  packages: [ package ]

Environments and deployment strategies

An environment is a collection of resources that can be targeted for deployment from a pipeline. Environments can include Kubernetes clusters, Azure web apps, virtual machines, databases etc. Typical examples of environment names are Dev, Test, QA, Staging, and Production.

Environment view

Learn more

Kubernetes and Virtual Machine resources in environment

Kubernetes resource view provides the status of objects within the mapped Kubernetes namespace. It overlays pipeline traceability on top so you can trace back from the Kubernetes object to the pipeline and to the commit. Virtual machine resource view lets you add your VMs that are on any cloud or on-premises. As a result, rollout application updates and track deployments to the virtual machines.

Resources in environment

Approvals and checks on resources

Pipelines rely on resources such as environments, service connections, agent pools, and library items. Checks enable the resource owner to control whether a stage in a pipeline can consume the resource. As an owner, you can define the checks that must be met prior a stage consuming that resource can start. For example, a manual approval checks on an environment ensures that deployment can happen only after the reviewers have signed-off.

Approvals and checks on resources

Learn more

Review apps for collaboration

Review apps works by deploying every pull request from your git repository to a dynamically created resource in an environment. The team can see the changes in the PR, as well as how they work with other dependent services before they’re merged into the main branch. As a result, you to shift-left the code quality and improve productivity.

Review apps for collaboration

Learn more

Refreshed UX for service connections

We have improved the service connection user experience. Further, enabled configuring approvals and checks on service connections. For approvals, we follow segregation of roles between the resource owners and developers. For instance, any pipeline runs that use a connection with the checks enabled will pause the workflow.

Refreshed UX for service connections

Learn more

Thank you

Lastly, we want to thank all our users who have adopted Azure Pipelines. If you’re new to Azure Pipelines, get started for free on our website.. Learn what’s new and what you can do with Azure Pipelines. As always, we’d love to hear your feedback and comments. Feel free to comment on this post or tweet at @AzureDevOps with your thoughts.

48 comments

Leave a comment

  • Avatar
    Janusz Nowak

    Nice that one for GA Multi-stage YAML pipelines (for CI and CD), any plans for allowing to chose version of resources pipelines when scheduling Release in Yaml or option to have possibility to use variables to select version of resources, missing also option to have environment deployment on higher level then job ?

    • Avatar
      Ben Duguid

      You can use the Library to create groups of variables, and then load those at different points in the YAML – the pipeline as a whole has a variables section that you can load a library set into, and then each stage can also load in variable groups scoped as you would expect.

      The editing experience isn’t anywhere as good as the scoped variables on a release pipeline, with lots of copy and paste and chances for errors though 🙁

    • Avatar
      ConsulTent Support

      There are actually three places where variables may be stored now:
      1. Directly in the Pipeline yaml code.
      2. In a Variables sections when you edit the pipeline in your browser. This is actually a neat feature that allows you to override the variables with the override option set, per manual run.
      3. The Library. You will want to use this if you have secret keys, or if you want to link with the Azure Vault.

      Using a combination of all 3 allows a lot of flexibility.

  • Avatar
    Michael Taylor

    I assume this still does not support the 1 feature that everybody that uses “classic” releases is still waiting for – the ability to deploy to different environments using a single pipeline but not automated. Note that this is not the same as a manual approval and never has been. The team seems to keep lumping them together. The YAML pipelines require that all the stages either get run or opt out. In other words we build, deploy to Dev, maybe run tests, deploy to QA, maybe have a manual approval and then deploy to Prod. But the pipeline is “in progress” until all this completes. For companies that CD to Dev every day but only send a handful to QA maybe once a sprint and to Prod every couple of sprints you end up with either failed pipelines or multiple active ones.

    Is the highly requested feature of being able to use YAML pipelines and later (days, weeks even months) deploy to other stages using the same release like we can in classic or are we stuck using classic until MS realizes we don’t deploy the way you do?

      • Avatar
        Daniel Schroeder

        I totally agree with Michael. This is the single feature that is keeping us from moving towards YAML for deployments. Sure, it can be accomplished via approvals, but that means constantly getting bombarded with emails everyday for every PR that gets merged in, when really we only want to push one version to Prod each sprint. Using approvals is a very clunky work around. I just want every PR merged in to get pushed to Dev and Int automatically, but want to manually push a button to go to RC and Prod, without having to tell everyone in the company to setup special email rules to send all approval emails to their trash, effectively making the approval notifications worthless.

      • Avatar
        Michael Taylor

        @ConsulTent Support, this is in no way solved by multiple pipelines. In fact that would make it worse. With classic release I can go to any of my 80+ pipelines that I manage and see at a glance that release 1 of Prod A is currently in QA whereas release 2 of Prod A is in Dev. I cannot remotely imagine how that would look if I had a separate pipeline for each of my stages. Release pipelines are nothing more than glorified back of artifacts with deployment instructions. There is no reason why you would ever treat a deployment into an environment of the same artifacts as a brand new release, at least in companies I’ve been at.

        The other thing is that the pipeline is partially finished. A release isn’t official until it gets deployed to Prod. There can be any # of releases to Dev and likely less releases to QA that ultimately cumulate into a Prod release. At any point in time a QA release could be promoted to Prod. Meanwhile you could have multiple other releases in other stages. The release isn’t finished but the stage is. Multiple pipelines doesn’t solve the underlying issue is just replicates how many pipelines you’re dealing with.

        The ask from the community has always been to have the same ability as classic release in terms of deploying to stages on demand. Until YAML releases can do this there is no reason for anyone to complicate their existing, working releases by any hacky approach. If multiple pipelines work for you then great but it is not the solution to the problem.

        • Avatar
          James May

          @Michael: Thank you for posting this – it has saved me a heap of time of figuring this out myself!

          @Roopesh (author): it would be awesome if you included a link in the post to the roadmap/future plans, especially around feature parity. I’m really looking forward to when I can finally get my release definitions into git.

  • Avatar
    Daniel Schroeder

    Great to see all of the improvements. In the classic editor you had the ability to say “If this stage fails, redeploy the last successful version to the stage”. Is that feature planned for YAML pipelines? Also, a large area that feature currently lacks is that you can only rollback the same stage that failed. We often run integration or smoke tests as a separate stage, so being able to tell it to rollback other stages would be ideal. Thanks!

  • Avatar
    Marco Brouwer

    Great to see all of the improvements, but I also have something to add to the “cannot move forward unless”-list.

    Somewhere last year I had contact with the team about adding support for Release Gates. We need to create a change in ServiceNow for every new release to PRD, we did this using the “classic” pipelines but this is not yet available on YAML pipelines. This was supposed to be rolled out Q1 2020 but not sure what the new timeline is on this.

    Do you have any idea when this will be added? I want to move all our teams to YAML ASAP but not having this is blocking that.

  • Avatar
    Mick Anderson

    I have been looking at utilizing multi-stage yaml for the build and release. One of the hangups / questions that isn’t clear to me is on how to implement part of it as it appears the environment option is only available on a deployment job and not on the stage. What happens if you need to run multiple jobs on a deployment and have the approval aspects work for all the jobs? For example on the classic view you have the ability to have the approval before the stage is executed. Within that stage lets say you have 2 agent jobs and 1 deployment group job. How would that get setup within the yaml file?

    • Roopesh Nair
      Roopesh NairMicrosoft logo

      You can have any number of jobs, deployment jobs in a stage. Each of the deployment job can target a specific environment or different environments. At stage execution level all the checks from across all environments, service connections, agent pools etc will be enumerated and the workflow will pause at the stage boundary. That is the approval will apply for all the jobs even if one of them is using a protected resource. Hope it helps

  • Avatar
    Pedro Lamas

    Brilliant stuff! I actually thought this was already on “General Availability” as a month ago we moved away from Releases to full Yaml CD with shared templates that allow us to easily define a single deployment plan that gets reproduced on all environments!
    I just wish the expression support on the yaml templates was better (specifically, parameter type checking, array length, better null support and handling)

  • Chris Buttacavoli
    Chris Buttacavoli

    It doesn’t seem that the approval checks allows me to schedule the release for a later time, so the deployment happens immediately. Does this mean that I have to get up at midnight whenever I want to do all my PROD deployments? I am wondering if this is being worked on, and if so I’d appreciate a link to some more information about when it might be released. Can’t really go whole hog on this just yet for many of our critical systems until this feature is available.

    This is a great update and I like the direction this is heading.

  • Mark Kharitonov
    Mark Kharitonov

    This is cool. However, we cannot use it because it does not exist in the Azure DevOps Server 2019, i.e. the on-prem version. Can we move to the Services version? A complicated business, given the amount of the code and processes we have already.
    It is ironic that the most loyal customers of Azure DevOps who started using it when it was still TFS without a trace of presence in the cloud are now left behind with an obsolete technology.
    The feeling is that we are screwed for being loyal.
    Thank you very much for nothing.

  • Avatar
    Justin King

    Under the heading “Resources in YAML pipelines” a new resource has been listed: packages. However, no matter how much I bing/qoogle, I can’t find any reference/sample yaml on what the parameters are to use it. The best I can find is: here, but while it states packages can be used to trigger off azure artifacts (good), it’s also devoid of any sample structure.

    I tried guessing/trial-n-error, but came up short. It seems the pipeline accepts the resource but I cant figure out what params it wants. Any hints would be wonderful, been waiting for this feature for a long time (being able to separate deploy from build so one can select the artifact is a big deal).

    • Avatar
      Eric Smith

      Hey Justin I think I can help, try and use the YAML schema to view the properties\params.

      https://dev.azure.com/{OrgName}/_apis/distributedtask/yamlschema?api-version=5.1

      I found these searching for packageResource.

      "firstProperty": [
              "package"
            ],
            "required": [
              "package",
              "type",
              "connection",
              "name"
            ]