Removing older images in Azure Pipelines hosted pools

Whitney Jenkins

Whitney

Over the past year, we have been able to update Azure Pipelines hosted images faster than ever before. We have also rolled out new images – Windows Server 2019 with Visual Studio 2019, Ubuntu 18.04, and macOS Mojave 10.14. We have seen a phenomenal growth in the use of hosted pools.

To prepare for another set of exciting updates this year and to better serve the needs of our growing user base, we have to make room – both in terms of Azure capacity and in terms of human support resources – and remove some of the less used, older images. On March 23, 2020, we’ll be removing the following Azure Pipelines hosted images:

  • Windows Server 2012R2 with Visual Studio 2015, a.k.a vs2015-win2012r2 or Hosted
  • macOS X High Sierra 10.13, a.k.a macOS-10.13
  • Windows Server Core 1803, a.k.a win1803

This blog outlines the steps you must take now if you use one of these images.

Schedule changes

Based on user feedback and higher usage of the vs2015-win2012r2 image, we have decided to delay removal of the vs2015-win2012r2 image until March 31, 2020. Pipelines using the Hosted agent pool will be automatically redirected to use the vs2017-win2016 image.

Current Status

As of March 31, 2020 the following Azure Pipelines hosted images have been removed.

  • Windows Server 2012R2 with Visual Studio 2015, a.k.a vs2015-win2012r2 or Hosted
  • macOS X High Sierra 10.13, a.k.a macOS-10.13
  • Windows Server Core 1803, a.k.a win1803

In addition to removing these hosted images, the Hosted agent pool has been redirected to use the vs2017-win2016 image.

Windows Server 2012R2 with Visual Studio 2015 (vs2015-win2012r2 or Hosted)

We have decided to remove the vs2015-win2012r2 image because it has not been updated in several years, lacks various security patches and hotfixes, and less than 5% of our users use it. This includes the agent pool labeled Hosted. While we have decided to remove the vs2015-win2012r2 image, there are two newer versions of Visual Studio images available in Azure Pipelines – vs2017-win2016 and windows-2019.

We understand that some of you still build your older apps using this image. Here are your options to move forward:

  • Identify pipelines using the vs2015-win2012r2 image, by navigating to https://dev.azure.com/{yourorganization}/{yourproject}/_settings/agentqueues. From there, you will need to check both the Azure Pipelines and Hosted agent pool for pipelines impacted by this change. Please note, in the Azure Pipelines pool, you will need to use the agent specification dropdown to filter for vs2015-win2012r2. In addition, you can use the query job history for retired images script.

The pipeline name is underneath the job (e.g. PartsUnlimited (2) is the pipeline)

  • Use one of the newer versions of Visual Studio, if that is an option for you. In many cases, your apps can be migrated to build on a newer version of Visual Studio with minimal effort. The vs2017-win2016 and windows-2019 images are kept up to date. If you use classic pipelines, edit your pipeline and change the image to one of these. If you use YAML pipelines, update your pipeline by editing its YAML file.
    jobs:
    - job: Windows
      pool:
        vmImage: 'vs2017-win2016'
      steps:
      - script: echo hello from Windows
  • Run your builds on a self-hosted agent pool. You can use Azure to create new virtual machines in the cloud, and to install Visual Studio 2015. Once created, configure a self-hosted agent pool in your Azure Pipelines organization, and configure each of your pipelines to use the self-hosted agent pool.

macOS High Sierra 10.13 (macOS-10.13)

We have decided to remove the macOS-10.13 image because less than 5% of users use macOS High Sierra 10.13 and there is a later macOS version image – macOS-10.14 available. In addition, we will be adding the latest macOS version – Catalina or macOS-10.15 on February 3, 2020.

This means that versions of Xcode older than 9.4.1, will no longer be supported. If your builds run on an older version of Xcode, please take some time to update to Xcode 10 or later to continue taking advantage of the Hosted macOS images. You can also set up a self-hosted macOS agent pool with your desired tools and configure your pipelines to use that pool.

To identify pipelines using the macOS-10.13 image, navigate to https://dev.azure.com/{yourorganization}/{yourproject}/_settings/agentqueues. From there, you will need to check both the Azure Pipelines and Hosted macOS High Sierra agent pool for pipelines impacted by this change. Please note, in the Azure Pipelines pool, you will need to use the agent specification dropdown to filter for macOS-10.13. In addition, you can use the query job history for retired images script.

The pipeline name is underneath the job (e.g. PartsUnlimited (2) is the pipeline)

If you have specific questions regarding our Hosted macOS pools, please reach out to Nilofer on Twitter at @nilli_minaj

Windows Container (win1803).

The usage of this image is the lowest among our pools – less than 1%. Any builds that target this pool will start failing on March 23, 2020.

To identify pipelines using the win1803 image, navigate to https://dev.azure.com/{yourorganization}/{yourproject}/_settings/agentqueues. From there, you will need to check both the Azure Pipelines and Hosted Windows Container agent pool for pipelines impacted by this change. Please note, in the Azure Pipelines pool, you will need to use the agent specification dropdown to filter for win1803. In addition, you can use the query job history for retired images script.

The pipeline name is underneath the job (e.g. PartsUnlimited (2) is the pipeline)

To mitigate this:

Brownout schedule

During the next couple of weeks, we will conduct a series of brownouts to highlight builds and releases using these soon-to-be-removed images. We will conduct our first brownout on March 9th from 1800-1900 UTC. After that we will continue to fail builds that use these images for up to 3 hours every day at different times. You will see the following error message in your build or release, if you are affected by a brownout:

This job was temporarily blocked because it uses a Microsoft hosted agent () that will be permanently removed on March 23, 2020. See https://aka.ms/blocked-hosted-agent for more information.

During these periods, any builds and releases that use vs2015-win2012r2, macOS-10.13, or win1803 images will fail. After each brownout period, we will revert the change. If your builds and releases fail during these brownout periods and if you see the error message shown above, please update your image to one of the supported images highlighted in this blog post.

Feedback

We’d really like to hear from you regarding what capabilities you’d like to see us add to the Hosted Agents experience in Azure Pipelines. Please contact us by reaching out @AzureDevOps or by creating an issue on our GitHub repository.

Thank you for using Azure Pipelines!

30 comments

Comments are closed. Login to edit/delete your existing comments

  • Avatar
    Murphy, William W

    Anyway to get a sample for using PowerShell,RestAPI, or other method to check the classic build and release definitions for references to these agents. We have some pipelines that have been executed outside the history window of the Agent Queue. It would be helpful in identifying which ones are important enough to update and which ones might need to be removed due to inactivity.

  • Avatar
    Jitin JohnMicrosoft employee

    Sad to see Windows Container (win1803) being removed. I was using this heavily since it is the lightest and least used agent. I am guessing usage of vs2019 agents are high even for tasks that do not use VS and may have very simple tasks that can be done in containers.

  • Avatar
    Victor Xavier

    I don’t know if there is any connection, however after this change, my pipelines are in the status queued “Waiting for a job.” until time out returns and cancels the process. Do you know what may be happening?

  • Avatar
    Greg Veres

    This document doesn’t really tell me how to move my release task from vs2015-win2012r2 to a newer version. All it explains is how to find the offending pipeline, which I already knew because I saw the warning in the logs. I can’t find where to change from vs2105-* to something newer. Please consider updating this document with instructions on how to make the actual change that is needed. Thanks.

    • Avatar
      Greg Veres

      I figured it out after some experimentation. I have been using Azure Dev Ops for 4 years now and the tasks were setup quite a while ago, which meant they were using “Hosted” as the pipeline, not the new “Azure Pipelines”. Hosted has no options when editing the Agent Job. You have to select “Azure Pipelines” to be able to see the list of pipeline choices. From there it is straight forward to select the proper one.

      However, there doesn’t seem to be a VS2019 enabled pipeline host. I don’t know what “windows 2019” means. I assume it is just a newer version of windows with no Visual Studio installed.

    • Avatar
      Титков Владимир Сергеевич

      Hello. Does anybody cares about SharePoint 2013 development in a Microsoft company? Your other images doesn’t work with SP 2013! You can’t just shut the single working agent down!

  • Avatar
    John Huang (POWERAPPS)Microsoft employee

    Whitney, we are not calling any specific agent pool or containers, such as the “vs2015-win2012r2”. We are just calling the generic “Hosted” pool, yet we get the brownout message and all of our builds failed during the experimental window yesterday. Is that expected?

    I thought that by following your example and making sure we weren’t explicitly calling the set of agents that were about to retire was sufficient. Did we have to update all of our build defs (yaml) to use the “windows-2019” agent pool?

    By the way, does the “windows-2019” agent pool contain any version of Visual Studio?

    • Avatar
      Todd Girvin

      Same here. We’re getting brown-outs on the Windows Hosted Pool and when I choose Azure Pool with the “windows-2019” agent. Is there something wrong with the brown-out logic?

    • Avatar
      Nathaniel KatzenbergerMicrosoft employee

      +1 Builds in our ‘Hosted’ pool are failing, and there is no option to change the agent specification. Do we need to move all of our pipelines out of this pool or is this just a bug that will age out?

      • Whitney Jenkins
        Whitney JenkinsMicrosoft employee

        Hi All,

        If your pipeline is listed as using the Hosted agent pool. Select Azure Pipelines from the agent pool drop down menu. Once Azure Pipelines is selected, you should see a new drop down titled agent specification. This will give you a list of images to choose.

          • Avatar
            Jeff MendozaMicrosoft employee

            Thanks for the clarification and updating the blog. I didn’t know if “Hosted” would just get automatically upgraded, since there is no selection.

            “If you use classic pipelines, edit your pipeline and change the image to one of these.” This is very terse. It would be great to expand this with some screenshots. Imagine someone new to Azure DevOps took over a bunch of old build that are now failing!