Safely Upgrade Your Pipelines from Azure DevOps Server 2019 to Server 2020

Silviu

Senior Product Manager

Azure DevOps Server 2019 and 2020 use two different models for retaining builds. In Azure DevOps Server 2019, users configure retention settings for each pipeline. Organization-level defaults are applied if a pipeline is not explicitly configured for retention. Builds that are retained indefinitely or by releases are marked and handled using a common bit in the build record. It is hard to infer why a build is being retained. If it is being retained by a release, it is hard to infer which release is retaining it.

To address these challenges, we overhauled the retention model in Azure DevOps Server 2020. Project-level retention settings are introduced. Moreover, users protect runs from automatic deletion by creating “retention leases” either manually or through these project-level retention settings. If the same build is being retained for two reasons, there would be two leases on that build – each explicitly capturing information about why it is being retained. As long as a build has at least one valid lease, it will not be removed.

In order to bridge the two models, Azure DevOps Server 2020 handles build retention using a hybrid model:

For pipelines without pipeline-level retention policies, it uses the new retention model.
For pipelines with pipeline-level retention policies, it uses the new retention model if those policies match the organization-level defaults. Otherwise, it uses the old model.

Over the past few months, we have learned from some customers about retained builds unexpectedly being deleted after an upgrade from Azure DevOps Server 2019 to 2020.

If you have a pipeline with per-pipeline retention settings (that were overridden from the organization level defaults) in Azure DevOps Server 2019, then these pipelines will be fine after the upgrade to 2020. All of its builds that are retained (either manually or by release or by policy) prior to the upgrade will continue to exist after the upgrade.

But, if you have a pipeline without per-pipeline retention settings (or if you have per-pipeline settings that exactly matched the organization defaults) prior to the upgrade, then these builds may be deleted after the upgrade since they are governed by the new retention model. If the build is:

Marked explicitly as retained by default, then it will still be retained after the upgrade.
Retained by a release, then it will still be retained after the upgrade.
Retained by the old retention policy, then it will be deleted after the upgrade.

Let us look at an example. Consider a Classic pipeline in Azure DevOps Server 2019 for which you did not modify the per-pipeline retention policy. In that version of the server, the default organization retention setting would apply. Let us say that according to that policy, the last 10 builds are being retained. You now upgrade to Azure DevOps Server 2020. The default project retention setting in Server 2020 is to retain the last 3 runs on each protected branch and to delete unretained builds older than 30 days. But, because your per-pipeline policy setting matches the default organization retention setting, all your builds older than 30 days will be deleted. If you produce your builds often, this is likely not a problem. But, if you have a pipeline that you use infrequently, then those 10 builds might be of importance to you.

To prevent the loss of builds when upgrading, follow the steps below.

Disable the new retention model

While still on Azure DevOps Server 2019, run the SQL script below. This will ensure that you do not switch to a hybrid model after the upgrade. Instead, you will still operate in the 2019 model.

USE [AzureDevOps_Configuration];
DECLARE @featureFlag    NVARCHAR(MAX)   = 'Pipelines.SimplifiedRetention'; -- the name of the feature flag
DECLARE @registryValue  NVARCHAR(MAX)   = '#\FeatureAvailability\Entries\' + @featureFlag + '\AvailabilityState\';
DECLARE @enable         BIT             = 0;
EXEC prc_SetRegistryValue 1, @registryValue, @enable;
-- check the result
SELECT  *
FROM    dbo.tbl_RegistryItems
WHERE   PartitionId > 0
        AND ParentPath LIKE '#\FeatureAvailability\Entries\' + @featureFlag + '%';

Make a backup of your Server 2019 database

Make sure to create a backup of your Azure DevOps Server 2019 database before upgrading to Azure DevOps Server 2020.

Upgrade to 2020 Server

Now that you have taken the precautionary steps, go ahead and upgrade to 2020 Server.

Create leases for important pipeline runs in Server 2020

You can manually create a lease for a pipeline run, or you can use the new Leases API to do it programmatically.

You can quickly see which runs have leases for them by navigating to a pipeline’s overview page and looking for the pin icon to the right of the commit hash.

Re-enable the new retention model in Server 2020

After you created leases for your important pipeline runs, you can turn on the new retention model, by running the following SQL script:

USE [AzureDevOps_Configuration];
DECLARE @featureFlag    NVARCHAR(MAX)   = 'Pipelines.SimplifiedRetention'; -- the name of the feature flag
DECLARE @registryValue  NVARCHAR(MAX)   = '#\FeatureAvailability\Entries\' + @featureFlag + '\AvailabilityState\';
DECLARE @enable         BIT             = 1;
EXEC prc_SetRegistryValue 1, @registryValue, @enable;
-- check the result
SELECT  *
FROM    dbo.tbl_RegistryItems
WHERE   PartitionId > 0
        AND ParentPath LIKE '#\FeatureAvailability\Entries\' + @featureFlag + '%';

Remove pipeline-specific retention policies

We recommend you remove per-pipeline retention policies and instead tune the per-project retention policy to your needs.

Conclusion

Azure Server 2020 introduces a new retention model. To safely upgrade from Server 2019 to 2020, without a loss of pipeline runs, we recommend you:

Disable the new retention model
Make a backup of your Server 2019 database
Upgrade to Server 2020
Create leases for important pipeline runs
Re-enable the new retention model
Remove pipeline-specific retention policies

FAQ

How does retention work when the new retention model is off?

In Azure DevOps Server 2019, no changes
In Azure DevOps Server 2020, the system will:
- Create leases for the most recent 3 builds generated while running Server 2020
- For YAML pipelines and Classic pipelines without per-pipeline retention policies:
  - Builds will be retained according to the collection-level maximum retention settings
- For Classic pipelines with custom per-pipeline retention policies:
  - Builds will be retained according to the pipeline-specific retention policy
- The builds with leases do not count toward the Minimum to keep setting

What happens to manually retained pipelines runs in Azure DevOps Server 2019?

They will be retained post upgrade to Azure DevOps Server 2020.

What happens to pipelines runs marked to be retained by a release in Azure DevOps Server 2019?

They will be retained post upgrade to Azure DevOps Server 2020.

This looks like a lot of work for an upgrade. Is there a better solution?

Yes, we are working on releasing another update for Server 2020 that does not have this issue. Please stay tuned for updates. If this procedure seems complex, then you can wait for that update to be released.

Author

Silviu

Senior Product Manager

15 comments

Discussion is closed. Login to edit/delete existing comments.

Sort by :

Newest

Newest Popular Oldest

Fokko March 9, 2022 0

You only mention the upgrade from Azure DevOps Server 2019, what happens when you upgrade from a previous (TFS) version? The same? And how to find out for which build definitions the retention has been set explicitly? When I use the REST API to retrieve the retentionRules property from a build definition, all build definitions have one set…
Hoysala G January 31, 2022 0
Thank you for this useful article! One of our customers was stuck in this issue for a long time; this article helped them move ahead. Few questions on what happens w.r.t migration to Azure DevOps from here -

Read more
Thank you for this useful article! One of our customers was stuck in this issue for a long time; this article helped them move ahead. Few questions on what happens w.r.t migration to Azure DevOps from here –
1. What build definition policies/settings do we need to put into place to migrate from 2020.1.1 to ADO Services, if any, in order to retain our builds for our projects the same way they are currently retained?
2. Once we get to ADO Services are there any special retention settings we need to set/update?
Read less
- Silviu Author February 14, 2022 · Edited 0
  Hi, Hoysala!
  
  Allow me to answer your questions inline.
  
  Read more
  Hi, Hoysala!
  
  Allow me to answer your questions inline.
  
  What build definition policies/settings do we need to put into place to migrate from 2020.1.1 to ADO Services, if any, in order to retain our builds for our projects the same way they are currently retained?
  
  Before migrating from Azure DevOps Server to Azure DevOps Service, please read and follow the instructions in our Migrate to Azure DevOps guide.
  
  Azure DevOps Services uses the new build retention model and does not support per-pipeline retention settings. For a smooth migration, you should ensure there are no per-pipeline retention settings.
  
  Regarding retaining builds, builds that already have a lease (they are, e.g., retained manually, retained by a release, retained by project-level retention settings) will be retained after the migration to Azure DevOps Services.
  
  As for the builds that do not fit in the list above and that you wish to retain, you should create leases for them, either manually or using the Leases API.
  
  Once we get to ADO Services are there any special retention settings we need to set/update?
  
  You should check the Project-level retention settings, as the maximum values of the settings for Azure DevOps Service may be lower than your on-prem settings. Be mindful that Azure DevOps Service no longer supports Organisation-level retention settings.
  Read less
sepehr mohammadi January 24, 2022 · Edited 0

Hi. Exactly When will RC1 be released?
amajsonic
- Silviu Author February 14, 2022 0
  
  Hi,
  
  We are working toward releasing Azure DevOps Server 2022 RC1, but I cannot confirm a release date or its scope just yet.
Best Quotes January 22, 2022 0

tanks for information.
best quotes
John King January 18, 2022 0

Is there a plan to upgrade to .net 6 , so we can simply deploy a local version inside docker like GitLab does ?
for now , Azure DevOps [server] is great, so let’s using GitLab 😂
- Silviu Author January 19, 2022 0
  
  Hi , John!
  
  I think the best place to ask about the support for .NET 6 is in the Developer Community.
dewo web January 18, 2022 0

Provide any information about Azure DevOps Server 2022. What features are included? When will RC1 be released? Please break the information on dewoweb
- Silviu Author February 14, 2022 0
  
  Hi,
  
  We are working toward releasing Azure DevOps Server 2022 RC1, but I cannot confirm a release date or its scope just yet.
James O'Malley January 18, 2022 · Edited 0

Thanks Silviu, I think you need to reassure us admins that running the posted SQL will not impact on any future support issues raised with Microsoft please.

Also, it would be great to be able to run something to get a report on the affected pipelines if possible
- Silviu Author January 19, 2022 · Edited 0
  Hi, James!
  
  Running the SQL commands described in this post, to turn off and then back on the feature, will not have an impact on any support issues raised with us. Our upcoming Azure DevOps Server 2020 update will essentially do the same thing, i.e., turn off the feature flag. Should you require support in the build retention area, we would be grateful if you included the status of this feature flag in your request.
  
  Your...
  Read more
  Hi, James!
  
  Running the SQL commands described in this post, to turn off and then back on the feature, will not have an impact on any support issues raised with us. Our upcoming Azure DevOps Server 2020 update will essentially do the same thing, i.e., turn off the Pipelines.SimplifiedRetention feature flag. Should you require support in the build retention area, we would be grateful if you included the status of this feature flag in your request.
  
  Your idea to get a report of the affected pipelines is great!
  
  We can help you figure out which are the pipelines that will use the new retention model after upgrading to Azure DevOps Server 2020. It is these pipelines’ builds that are older than 30 days that will be deleted when the feature flag Pipelines.SimplifiedRetention is enabled.
  
  What I suggest you do is run the SQL query below after you upgrade to Azure DevOps Server 2020 but before you enable the Pipelines.SimplifiedRetention feature flag. Then, you retain the builds that matter to you from the pipelines returned by the script. My hope is that not all your pipelines will be listed, so the number of pipelines you have to go through is reduced.
  
  USE [AzureDevOps_DefaultCollection]; DECLARE @minToKeepCollectionDefault INT = 1; -- the default Minimum to keep setting from Collection Settings -> Pipelines -> Retention -> Default retention policy SELECT d.DefinitionId, d.DefinitionName, d.RetentionPolicy, d.ProcessType FROM [Build].[tbl_Definition] d WHERE d.ProcessType = 2 OR -- YAML d.RetentionPolicy = '[]' OR -- No per-pipeline policy (JSON_VALUE(d.RetentionPolicy, '$[0].minimumToKeep') = @minToKeepCollectionDefault AND JSON_VALUE(d.RetentionPolicy, '$[1].minimumToKeep') IS NULL) -- A single per-pipeline policy where minimumToKeep matches minToKeepCollectionDefault
  
  Finally, please remember to turn back on the Pipelines.SimplifiedRetention feature flag after having retained your important builds.
  Read less
nikita egorov January 17, 2022 0

Provide any information about Azure DevOps Server 2022. What features are included? When will RC1 be released? Please break the information vacuum
- Silviu Author January 19, 2022 0
  
  Hi, Nikita!
  
  We are working toward releasing Azure DevOps Server 2022 RC1, but I cannot confirm a release date or its scope just yet.
  - Lukasz Dlugajczyk March 1, 2022 0
    
    So… Q1 is slowly closing to its end, and there is no RC or information about it. Is there any update on progress (i.e. you are no longer aiming at Q1)?