What’s new with Azure Pipelines

Avatar

Gopinath

Azure Pipelines, part of the Azure DevOps suite, is our Continuous Integration and Continuous Delivery (CI and CD) platform, used every day by large enterprises, individual developers, and open source projects. Today, we’re thrilled to announce new features for Azure Pipelines, including some much-requested ones:

  • Multi-stage YAML pipelines (for CI and CD)
  • Environments and deployment strategies
  • Kubernetes support

Multi-stage YAML pipelines

One of our biggest customer requests since launching YAML support for Build pipelines (CI) has been to support it for Release pipelines (CD) as well. To accomplish this, we now offer a unified YAML experience, so you can configure each of your pipelines to do CI, CD, or CI and CD together. Defining your pipelines using YAML documents allows you to check the configuration of your CI/CD into source control together with your application’s code, for easy management, versioning, and control.

Multi-stage pipelines view

With our new YAML support, we’re also bringing a new UI to help visualize all of your multi-stage pipelines across the product, whether you’re in the run summary view, looking at all your pipeline runs, or browsing logs.

View all runs

In addition to our new pipelines pages, we have a new log viewing experience as well. This lets you easily jump between stages and jobs along with helping you quickly identify errors and warnings.

New logs view

This feature will be rolled out for all accounts over the next few days. To enable it, go to the preview features page and turn on the toggle for “Multi-stage pipelines”.

Getting going with YAML

We want you to be able to get going fast wherever your code lives. Once you connect your repo, whether it’s on GitHub, Azure Repos, or your own external Git source, we’ll analyze your code and recommend a YAML template that makes sense for you and gets you up and running quickly.

Configure pipeline from template

While we want to get you running quickly, we know you’re going to want to keep configuring and updating your YAML. To help make it even easier to edit and update your pipeline, we’ve created an in-product editor with IntelliSense smart code completion, and an easy task assistant.

YAML editor with IntelliSense

Building your first multi-stage pipeline with environments

Bringing CD to YAML means a bunch of great additions in terms of commands and functionality. Let’s cover the basics with a simple pipeline that just builds and deploys an app in two stages.

stages:
- stage: Build
  jobs:
  - job: Build
    pool:
      vmImage: 'Ubuntu-16.04'
    continueOnError: true
    steps:
    - script: echo my first build job
- stage: Deploy
  jobs:
    # track deployments on the environment
  - deployment: DeployWeb
    pool:
      vmImage: 'Ubuntu-16.04'
    # creates an environment if it doesn’t exist
    environment: 'smarthotel-dev'
    strategy:
      # default deployment strategy
      runOnce:
        deploy:
          steps:
          - script: echo my first deployment

If we ran this pipeline, it would execute a first stage, Build, followed by a second stage, Deploy. You are free to create as many stages as you wish, for example to deploy to staging and pre-production environments.

You may notice two new interesting concepts in here if you’re familiar with our YAML schema. And if this is the first time you’re seeing our YAML, you can read up on the core concepts here.

The first new keyword is environment. Environments represent the group of resources targeted by a pipeline, for example, Kubernetes clusters, Azure Web Apps, virtual machines, and databases. Environments are useful to group resources, for example under “development”, “staging”, “production”, etc, and you can define them freely. Defining and using an environment unlocks all kinds of capabilities, for example:

  • Traceability of commits and work items
  • Deployment history down to the individual resource
  • Deeper diagnostics, and (soon) approvals and checks

There’s a lot of great new functionality available today in preview, and even more coming around the corner. You can learn more on environments here

Kubernetes environments

You will also notice the strategy keyword. This impacts the deployment strategy, which defines how your application is rolled out across the cluster. The default strategy is runOnce, but in the future you’ll be able to easily indicate other strategies, such as canary or blue-green.

If you’re ready to start building, check out our documentation for building a multi-stage pipeline with environments. If you want to see some multi-stage pipeline templates to work off of, take a look at out our templates repo. You can even see those sample pipelines in action inside of our samples project.

Kubernetes

If you have an app which has been containerized (ie. there is a Dockerfile in the repository), we want to make it easy for you to setup a pipeline in less than 2 minutes, to build and deploy to a Kubernetes cluster (including Azure Kubernetes Service). Wrapping your head around Kubernetes can be hard, so we’re making it easy to both get started and continue deploying to your Kubernetes clusters. For more details, read our post on Azure Pipelines and Kubernetes.

Kubernetes is fully integrated with Azure Pipelines environments too. This lets you view all the deployments, daemonsets, etc, running on Kubernetes in each environment, completed by insights such as readiness and liveness probes of pods. You can use this information and pod-level details (including logs, containers running inside pods, and image metadata) to effectively diagnose and debug any issue, without requiring direct access to the cluster itself.

Kubernetes environments

A look at what’s next

In addition to the preview features that now available, there are so many exciting things just around the corner for Azure Pipelines we want to share:

  • Caching – We’ll be announcing the availability of another much-requested feature very shortly: caching to help your builds run even faster.
  • Checks and approvals – We’re improving multi-stage pipelines with the ability to set approvals on your environments, to help control what gets deployed when and where. We’ll keep iterating here to deliver more experiences with checks to help gating your multi-stage pipelines.
  • Deployment strategies – We’re adding additional deployment strategies as part of the deployment job type, like blue-green, canary and rolling, to better control how your applications are deployed across distributed systems.
  • Environments – We’re adding support for additional resource types in environments, so you can get going quickly with virtual machines (through deployment groups) and Azure Web Apps.
  • Mobile – With our new UX, we’re going to start to enable new mobile views in Q2 to help you view the status of pipelines, quickly jump into logs, and complete approvals.
  • Pipeline analytics – We’re continuing to grow our pipeline analytics experiences to help you get an all-up picture of the health of your pipelines, so you can know where to go in and dig deeper.
  • Tests and code coverage – We’re going to be shipping all new test and code coverage features and UX in the next months.

Thank you

Lastly, we want to thank all of our users who have adopted Azure Pipelines. Since our launch last September, we have seen tremendous growth, and we are particularly excited about the response from the open source developer community. It’s been an honor to see Azure Pipelines badges on so many open sources projects we love and use regularly ourselves. In the first eight months, public repositories have already used over 88 years of build time for Azure Pipelines for free. Check out Martin’s post for some more stats and stories from open source projects. We’ve also received so much great feedback from project maintainers to date and we can’t thank the community enough.

If you’re new to Azure Pipelines, get started for free on our website and learn more about what you can do with Azure Pipelines.

We’ll be posting more deep-dive information in the coming weeks, but as always we’d love to hear your feedback and comments. Feel free to comment on this post or tweet at @AzureDevOps with your thoughts.

Avatar
Gopinath Chigakkagari

Group Program Manager, Azure Pipelines

Follow Gopinath   

61 comments

  • Avatar
    Lachlan White

    How do i turn this on? The preview feature isn’t available in my Azure DevOps Organization? Is this only enable in specific regions currently?

    • Avatar
      Christopher Haws

      The post states: “This feature will be rolled out for all accounts over the next few days. To enable it, go to the preview features page and turn on the toggle for “Multi-stage pipelines”.”
      I have not received the preview feature yet either. 🙂

    • Avatar
      Van Caloen Jean

      Great news, we’ve been waiting on yaml release for a long time!
      It’s still not available for us. Any timeline for the rollout to Europe ? And for the approvals, any timeline for the activation ?

    • Avatar
      Andrew May

      Also the YAML schema is missing deployment, environment etc. There’s a few examples scattered amongst the rest of the documentation, but nothing comprehensive like the schema.
      I had really hoped that deployment would support containers, but that does not appear to be the case (at least yet).

  • Avatar
    Erik Ammerlaan

    Very cool! Our team had a build pipeline using YAML producing artifacts, and a classic release pipeline that deployed them. I succesfully converted this today into one multi-stage YAML pipeline.Only thing that I’m missing so far, to be able to use it for real, is a manual approval step between staged. I tried out the ManualIntervention task on a serverless job, but as expected, it doesn’t work yet. Can’t wait for approval steps to be there too! Keep up the good work.

    • Tim Jacomb
      Tim Jacomb

      Any chance you have an example?
      I’m trying to have a build and release stage which has deployments in it, but my files from the repo don’t seem to be available

      • Avatar
        Erik Ammerlaan

        The deployment job will not checkout your repo. Instead, use a PublishPipelineArtifact task in your build stage to package the files you want to make available for deployment and then use a DownloadPipelineArtifact task in your release stage to get them.Great thing about this is that if you have multiple stages (like Development, QA, Production), those stages can use exactly the same deployable artifact, which makes your pipeline more reliable.

  • Avatar
    John Hartley

    I’m confused about whether this is enabled for my organization or not. I can specify a deployment job and environment, but it says none of my deployment groups can be found
    When I use an agent pool the pipeline artifact download for the deployment always fails with a hard linking error

  • Avatar
    Jorrit Meijer

    Looks like your missing an external reference in this sentence:
    There’s a lot of great new functionality available today in preview, and even more coming around the corner. You can learn more on environments here
    here actually doesnt go anywhere.

  • Avatar
    Russell McCloy

    Hi there.
    Just been working through a POC of setting all this up. It’s pretty good but couple of things: – when your is a validation error in your yaml it stops you running your pipeline. This is fine apart from the notification that something is wrong is empty. Took me a while to work out what was blocking me, Found the real validartion error in source in browser
    – secondly, I cant seem to pass things between stages. If I have a build stage that includes a PublishBuildArtifacts@1 and then I have a deploy stage that will deploy arm templates and code to an Azure web app, (AzureResourceGroupDeployment@2 and AzureRmWebAppDeployment@4). These two deployment stages can see the build artifacts. If I move every to a single stage, it all works. Maybe artifacts arent available from stage to stage but I cant find mention of this in the documentation?
    thanks
    Russ

    • Avatar
      Erik Ammerlaan

      I had the same. You have to change PublishBuildArtifact to PublishPipelineArtifact to make it work. And then use DownloadPipelineArtifact in the release stage.

  • Avatar
    Grote, Justin

    “One of our biggest customer requests since launching YAML support for Build pipelines (CI) has been to support it for Release pipelines (CD) as well. To accomplish this, we now offer a unified YAML experience, so you can configure each of your pipelines to do CI, CD, or CI and CD together.”
    Except Release Pipelines still can’t be configured by YAML.
    Don’t get me wrong, multi-stage is a great improvement for build pipelines, but this announcement made me thing you had added YAML support for release pipelines, and unless I’m missing something, you haven’t. Release Pipelines still have the same old UI that isn’t YAML-able, and things like approvals and gates can’t be done in Build Pipelines.
    Is there still a roadmap to get Release Pipelines to be YAML as well?

    • Avatar
      Johanning, Joshua P.

      I too would like to express my disappointment in the phrasing of this announcement. As stated, this looks like the ability to have multi-stage for Builds and Releases, but what this looks like is a glorified multi-stage Build pipeline. I would prefer to have my build YAML separate from my Release YAML. 

  • Avatar
    James Dumont

    Amazing, thanks for all those improvment! I was doing some test and I couldn’t figuring out how to set through YAML a run trigger for new Artifacts or for a determined scheduled (for scheduled I found how to dot it through UI only). Is this someting that we can do during the preview or is this planned?

  • Avatar
    Jeff Chen

    Thanks for a great post, many useful information. Looking forward to hear more about Azure Pipelines. ‘Everything as Code’ is the future. Therefore, supporting existing pipeline capabilities(task and configuration) and easily migrating existing pipeline to YAML foramt are very crucial for entreprise customer.
    I am more keen on the ‘Environment’. We have multiple deployment groups and each of them contains one or multiple target machine(s). Currently, the limitation of this deployment group is that I cannot easily get the ‘tag’ of each target marchine during deployment. Hope you can improve this situation during development of ‘Environment’.
    Second thing is that it would be great to allow seperation of Build and Release pipeline in YAML too. We have situation like we need same build artifact can be deployed by multiple different release pipelines, which same the time of repeat building.

    (why there is no blank line between paragraphs after saving the comment ?)

    • Avatar
      Gopinath Chigakkagari

      We will be building build/release separation support even in YAML as well – we will be enabling pipeline artifacts and triggers soon.

      On the Environments – we will be adding Deployment Group into Environment as well. Keep sharing your feedback.

      • Avatar
        Hossam M. Hammady

        Is there an ETA for deployment groups? We have customers running behind proxies and internal machines that we would like to integrate multi stage pipelines to deploy to them. We can’t migrate because of that.

  • Avatar
    Satheesh Antonyraj

    Just one doubt, how to do I stop after deploying to test and wait for manual trigger to start the deployment to prod stage, i dont find any triggers in your yaml

  • Venura Athukorala
    Venura Athukorala

    Adding YAML support to the deploymnets would have made more sense. But, i can think of a lot of practical reasons not to go down that path. 
    Reinvent of CD for YAML means it will take a long time to mature. 
    Well, for now let’s please start with adding approval gates support (allowing the use of built in security groups)

  • Avatar
    Dennis van Emmerik

    Would it also be possible to set the variables per stage like in Release Pipelines now?One team here is using this, especially the grid-view of variables/stages to keep track of all the settings. 
    The stages will probably be then the same as the ‘environment’-setting that was added.

  • Avatar
    Paul Hines

    When will WebApps be supported?
    And is this to replace the normally seperated releases?
    Also how will it deal with multiple builds that have multiple artifacts that need to go into a single release? 

  • Avatar
    Alexander Trauzzi

    I’m not so sure I like the idea of hard-coding environment names into source control, seems like a step backwards.  Environments should be parameters to a pipeline, not written inside of them.

  • Gousiya Sayyad
    Gousiya Sayyad

    Hello Gopinath Chigakkagari,    Have any idea on the previous Version of the Build instead of the latest image automatic deploy in k8s cluster?    Our requirement is just rollback concept rather than all-time latest images

  • Avatar
    Richard Gavel

    In a release pipeline, I can use the artifact from my build pipeline, but I can also make use of artifacts from other pipelines (in my case, configuration settings files to be utilized in deploying the build artifact. How can I add a download a built artifact from another build?

  • Avatar
    W Tahara

    Gopinath, what are your thoughts on how multi-stage YAML pipelines impact branch policies (e.g. build validation/build policy). Does it make sense to use a multi-stage YAML pipeline for build validation which will also deploy an application? Why or why not?

  • sandhya chandrasekaran
    sandhya chandrasekaran

    I am trying to configure both CI and CD as yaml, CI is working as expected but CD’s – release predefined variable is not working with Build pipeline.
    Please let me know is it possible to configure CD as yaml

  • Avatar
    Mike Murray

    We have a build pipeline using YAML producing artifacts, and a classic release pipeline that deploys them.  I’d like to convert the classic release pipeline into YAML and integrate it into the existing build pipeline YAML file.  These pipelines are very custom, with each having different triggers.  I’d like to view the YAML for the entire release pipeline, but I see no way.  Nor do I find an updated definition of the YAML schema available for the release pipeline concepts.  Exporting the release pipeline produces a 350 line json file, but this is not much help without YAML schema.  If the YAML schema documention is/was complete, I think it would still be difficult to do the conversion.
    Is there more guidance available for this type of conversion process?  Why can’t we view the entire classic release pipeline as YAML?
    Thanks, Mike

  • Avatar
    Shanoor HASSANALY

    Environmnet variables are not injected to bash/AzureCLI tasks in deployment jobs, which makes them completely useless. Is it a bug or not yet implemented?

  • Avatar
    Jacques du Plessis

    Anyone experiencing their “Pipelines view” does not show the last built status? I can see it when i click on “Runs” but not in the “Recent” tab

  • Avatar
    River Yang

    Just want to know when will this be available on Azure DevOps Server 2019?
    I’m running Azure DevOps Server 2019.0.1 Release Date: May 21, 2019
    I’m not able to see all these changes… 
    Thanks!
    River

  • Avatar
    Michael Withagen

    Great improvement although I was hoping I could configure my releases in a separate yml file. This will at least improve the maintainability of my releases.

    Question: (When) will it be possible to redeploy to an environment?
    In my current setup my builds produce artifacts and they get released to different environments. When a rollback is needed I can go to any previous version and redeploy to any environment needed with existing artifacts, no rebuild required. This is quick and stable, it will guarantee a certain quality. 

  • Niels Swimberghe
    Niels Swimberghe

    One of the screenshot shows ” – deployment: DeployWeb”, but I don’t see “deployment” as a keyword in the YAML Schema reference.Where can we learn more about this keyword?

  • Gerome Ren
    Gerome Ren

    everything is done and cool on end.. the only thing is lacking manual approval is really a pain.. if there is any workaround that> i have to setup multiple pipelines for different environments…..

  • Avatar
    Mykyta Shevelov

    Hi Guys,
    We are developing a plugin for Azure DevOps under the summary view on build results/details view with target type:
    ms.vss-build-web.build-results-summary-tab
    and it works fine but I don’t see our plugin when Multi-stage pipelines is turned on.
    Can someone help to define which target type or contribution id we should use?

  • Avatar
    Hemanth Kumar Vanapalli - C51236

    I would like to run the Azure Pipelines by configuring Ant.

    However, while making changes to yaml, file, I am not able to refer the encrypted variables from Pipeline in the build.xml or build.properties files.
    How could I refer the Pipeline variables in build.xml or build.properties.

    Thanks

Leave a comment