This post begins by laying out the multi-repository world that makes up the .NET product, its inherent challenges, and how we deal with them. This is a review of some of the information presented in\u00a0The Evolving Infrastructure of .NET Core<\/a>. Then it takes a close look at how we build, prep, and ship the product, especially around releases that include security fixes.<\/p>\n .NET is developed not as a monolithic repo, but as a set of repos that have inter-dependencies on one another. See\u00a0.NET repositories<\/a>\u00a0for information on where various functionality is developed. For instance, the\u00a0 The upshot of this is that when we release a version of .NET, we cannot simply produce a build of each repo in parallel, sign, publish and release. Instead, we must ensure that the desired version of each dependency is referenced in all repos that make up the product and reference that dependency. Updating a version of a dependency in a repo means creating a new build against those updated dependencies, which in turn may require other repos to update their dependencies and rebuild. This set of inter-dependent repos forms a graph. In .NET 5, this dependency flow graph is currently 6 layers deep ( Given that this dependency graph is rather complex, we need automated ways to track and update it. We track dependencies via metadata in the\u00a0 The dependency names then correspond to a set of MSBuild property names present in an <\/span> The repo is then free to use the\u00a0 Notice that because the source information for a dependency is preserved within the\u00a0 We update our tracked dependencies automatically using a custom service called Maestro (see\u00a0https:\/\/github.com\/dotnet\/arcade-services<\/a>\u00a0for the implementation). Every official build (on any branch) contains a stage that reports its status to the Maestro service. Maestro maintains a registry of metadata about the reported builds, including a list of their outputs, commits, repo urls, etc. Maestro then uses concepts called channels and subscriptions to determine what to do with these newly reported builds. We call this process ‘dependency flow’.<\/p>\n It’s also interesting to note that channels affect publishing of a build’s assets. A build not assigned to any channel does not publish its outputs anywhere. Assignment to a channel will trigger publishing of those assets, and the channel defines what the desired endpoints are. For instance, the channel for day to day development of .NET 5 indicates that files like Sdk installers or zip archives should be pushed to the\u00a0 One thing to note: currently the Maestro system defining channels, subscriptions and dependency flow does not have a publicly available portal for visualization, though Maestro’s behavior is publicly visible via dependency update pull requests. https:\/\/github.com\/dotnet\/arcade\/issues\/1818<\/a>\u00a0covers this feature. Let us know if this is valuable by commenting or voting on the issue.<\/em><\/p>\n Typical product development workflows assign build intent based on branch. As the number of repos and scales up, this model tends to break down. Different teams have different development practices and branching strategies motivated by varying requirements. This is especially true when repos ship in multiple vehicles that run on separate schedules or have differing servicing requirements. For example the Roslyn C# compiler ships in the .NET SDK as well as separately in Visual Studio. By using subscriptions which pull builds that have been to assigned to channels, we end up with a cleaner producer\/consumer model for the flow of dependencies.<\/p>\n For example, the\u00a0 Day to day, code gets checked into each repo, builds of those code changes are performed, and those builds are assigned to channels. Maestro uses the subscription information to flow new outputs produced by those builds to other repos. For development channels, this process occurs on a continual basis. The product is constantly evolving and changing. Because some repos flow through the dependency graph along multiple paths, there may be varying versions of a single dependency within the product at any given moment. We call this state “incoherent”. For day to day builds, this is usually fine. The output product generally behaves as expected, and teams can take their time reacting to breaking changes, introducing new functionality, or dealing with failures in dependency update PRs. When creating a product for release, however, we want a single version of each asset referenced within the repo dependency graph. A single version of the runtime, a single version of aspnetcore, a single version of each NuGet package, etc. We call this state “coherent”.<\/p>\n Incoherency represents a\u00a0possible<\/em>\u00a0error state. For an example let\u2019s take a look at the .NET shared framework runtime. It exposes a specific public API surface area. While multiple versions of it may be referenced in the repo dependency graph, the SDK ships with just one. This runtime must satisfy all of the demands of the transitively referenced components (e.g. WinForms and WPF) that may execute on that runtime. If the runtime does not satisfy those demands (e.g. breaking API change, bug, etc.), failures may occur. In an incoherent graph, because all repositories have not ingested the same version of the shared framework, there is a possibility that a breaking change has been missed.<\/p>\n Let’s now take a look at the workflow by which we build and release the product.<\/p>\n We can think of the Maestro subscriptions as forming a flow graph, each edge representing flow of changes between repositories. If no additional ‘real’ (not dependency flow) changes are made in any repo in the graph, then eventually the flow of changes will cease and the product will reach an unchanging, coherent state where there is a single version of each dependency. This is what we drive for on a regular basis for preview and servicing releases. The general flow for each release is as follows:<\/p>\n This process sometimes completes significantly in advance of a scheduled release date. That’s what we hope for. When that happens, the assets just wait in Azure blob storage and various package feeds, to be released on the scheduled ship date. Other times, this process completes very near the scheduled ship date, and there is very little breathing room. This is particularly challenging when the scheduled ship date is immovable, like when it is tied to the first day of a big conference.<\/p>\n Day to day development of .NET is done on GitHub. For releases containing security fixes, we need a non-public place to commit those fixes, combined with any non-security changes made publicly on GitHub. Committing and building internally prevents early disclosure of vulnerabilities that would place customer applications at risk. To satisfy this requirement, we maintain two parallel sets of branches in an Azure DevOps repo:<\/p>\n This parallel set of branches gives us a way to produce a product with security fixes while performing as many of our changes in the open as possible:<\/p>\n There is one unfortunate side effect of the public -> internal merges. If dependency flow is enabled on both public and internal branches, i.e. public builds flow to public GitHub branches and internal builds flow to internal branches, then every dependency flow commit on the public side would conflict as it is merged into the internal branches. This requires extensive, error-prone, manual intervention as there can be 10s of dependency flow commits\/day, spread across many repos. To avoid this, we largely turn off public dependency flow when building an internal release, except for some leaf nodes in the graph (e.g. roslyn, fsharp, MSBuild, etc.), and internal dependency flow when building a public release. Thus, the runtime and SDK functionality packaged with the daily builds of the publicly available installer falls out of date until release day. The .NET team will be looking into robust ways to maintain both public and internal builds in parallel in the future.<\/p>\n .NET divides the\u00a0outputs<\/strong>\u00a0of each build into two categories:<\/p>\n Typically, non-shipping assets exist for the purpose of inter-repo transport. For instance,\u00a0 There is one particularly interesting implication of our multi-repo OSS development environment. As discussed earlier in this deep dive, the repos tend to have strong dependency ties to one another. They redistribute many binaries and reference specific version numbers (e.g. NuGet package versions) from their dependencies. For example,\u00a0 The .NET team solves this by having 3 sets of NuGet feeds our build infrastructure:<\/p>\n The separation of shipping and non-shipping packages into different feeds is useful. Combining the required shipping package feeds with the standard\u00a0 Recently, a\u00a0developer noticed<\/a>\u00a0that they were unable to build the tagged\u00a0v3.1.300<\/a>\u00a0commit. The 3.1.300 SDK release was built internally, so the isolated “darc-int-” feeds automatically propagated into the\u00a0 After we have a coherent product with all desired fixes, we run it through an Azure DevOps pipeline which handles preparing a full product drop and handoff for additional validation. We utilize Maestro in an interesting way to do this. Starting at\u00a0 Our additional prep and validation is wide ranging, but includes activities such as:<\/p>\n Bugs or issues found in this stage are evaluated for severity and changes are made in the appropriate product repos if necessary. Otherwise, on to release!<\/p>\n Release day is largely about pushing the new product build to various endpoints (e.g.\u00a0Our Principles<\/h2>\n
\n
A Land of Many Repos<\/h2>\n
dotnet\/runtime<\/code>\u00a0repo builds the core .NET runtime and some additional NuGet packages. Its outputs are consumed by\u00a0dotnet\/aspnetcore<\/code>,\u00a0dotnet\/installer<\/code>,\u00a0dotnet\/extensions<\/code>, and a few others. Consumption of dependencies in .NET takes a few forms:<\/p>\n\n
dotnet\/aspnetcore<\/code>\u00a0may produce NuGet packages that encode a dependency on a specific version of\u00a0Microsoft.Extensions.Logging<\/code>.<\/li>\n\n
Microsoft.Extensions.Logging<\/code>\u00a0is released as a standalone NuGet package on\u00a0https:\/\/nuget.org<\/code>, it is also re-packaged within the\u00a0ASP.NET\u00a0Core shared framework as part of the dotnet\/aspnetcore build.<\/li>\ndotnet\/aspnetcore<\/code>\u00a0build can be installed as a standalone component, but is also contained within the .NET SDK that is produced out of\u00a0dotnet\/installer<\/code>.<\/li>\n<\/ul>\n<\/li>\ndotnet\/runtime<\/code>->\u00a0dotnet\/winforms<\/code>\u00a0->\u00a0dotnet\/wpf<\/code>->\u00a0dotnet\/windowsdesktop<\/code>\u00a0->\u00a0dotnet\/sdk<\/code>\u00a0->\u00a0dotnet\/installer<\/code>).<\/p>\nTracking our Dependencies<\/h3>\n
eng\/Version.Details.xml<\/code>\u00a0file of each repo. This file identifies the names and versions of a set of input dependencies. Each dependency also identifies the source commit and repo that was built to create it. For example, this excerpt is from\u00a0dotnet\/installer<\/code>. It identifies that\u00a0dotnet\/installer<\/code>\u00a0has an input asset called\u00a0Microsoft.NET.Sdk<\/code>\u00a0at version\u00a05.0.100-rc.1.20403.9<\/code>, produced out of a build of\u00a0https:\/\/github.com\/dotnet\/sdk<\/code>\u00a0at\u00a056005e13634b9388aa53596891bc2e8192e2978c<\/code><\/a>.<\/p>\neng\/Version.Details.xml (excerpt from\u00a0dotnet\/installer @ dc95de00046550c2a5a053153ca78ef89a114fd7<\/a>)<\/h6>\n
<?xml version=\"1.0\" encoding=\"utf-8\"?>\r\n<Dependencies>\r\n <ProductDependencies>\r\n <!-- Other dependencies.. -->\r\n <Dependency Name=\"Microsoft.NET.Sdk\" Version=\"5.0.100-rc.1.20403.9\">\r\n <Uri>https:\/\/github.com\/dotnet\/sdk<\/Uri>\r\n <Sha>56005e13634b9388aa53596891bc2e8192e2978c<\/Sha>\r\n <\/Dependency>\r\n <\/ProductDependencies>\r\n<\/Dependencies><\/pre>\n
eng\/Versions.props<\/code>\u00a0file that also lives in the repo:<\/span><\/p>\n<\/div>\neng\/Versions.props (excerpt from\u00a0dotnet\/installer @ dc95de00046550c2a5a053153ca78ef89a114fd7)<\/a><\/h6>\n
<PropertyGroup>\r\n <!-- Dependencies from https:\/\/github.com\/dotnet\/sdk -->\r\n <MicrosoftNETSdkPackageVersion>5.0.100-rc.1.20403.9<\/MicrosoftNETSdkPackageVersion>\r\n <MicrosoftDotNetMSBuildSdkResolverPackageVersion>5.0.100-rc.1.20403.9<\/MicrosoftDotNetMSBuildSdkResolverPackageVersion>\r\n <MicrosoftNETBuildExtensionsPackageVersion>$(MicrosoftNETSdkPackageVersion)<\/MicrosoftNETBuildExtensionsPackageVersion>\r\n <MicrosoftDotnetToolsetInternalPackageVersion>$(MicrosoftNETSdkPackageVersion)<\/MicrosoftDotnetToolsetInternalPackageVersion>\r\n<\/PropertyGroup><\/pre>\n
MicrosoftNETSdkPackageVersion<\/code>\u00a0property as it wishes. In this case, it’s used to specify the version of package named\u00a0<\/span>Microsoft.NET.Sdk<\/code>\u00a0as well as part of the path to an archive zip file produced in the\u00a0<\/span>dotnet\/sdk<\/code>\u00a0build.<\/span><\/p>\neng\/Version.Details.xml<\/code>\u00a0file, it’s possible to look up the\u00a0eng\/Version.Details.xml<\/code>\u00a0at the source repo + commit and determine\u00a0its <\/em>dependencies. Doing so recursively builds up a graph of all dependencies present within the product. See below for a visualization of a recent Preview 8 build. We divide these dependencies into two categories:<\/p>\n\n
https:\/\/nuget.org<\/code>) the product should function as expected.<\/li>\nUpdating our Dependencies<\/h3>\n
\n
main<\/code>\u00a0branch of\u00a0dotnet\/efcore<\/code>\u00a0may be intended for day to day development, meaning it’s outputs should flow to other repos’ branches that are also tracking day to day development. On the other hand, a build of a test branch is not intended to flow anywhere. Channels are effectively tags that signal\u00a0intent<\/strong>\u00a0for a build. Any build can be assigned to any channel.<\/li>\neng\/Version.Details.xml<\/code>\u00a0and\u00a0eng\/Versions.props<\/code>\u00a0files. It then\u00a0opens a pull request<\/a>\u00a0with the changes.<\/li>\n<\/ul>\n![\"ependency](\"https:\/\/devblogs.microsoft.com\/dotnet\/wp-content\/uploads\/sites\/10\/2020\/08\/Preview8DependencyFlow.png\")
<\/a>dotnetcli<\/code>\u00a0storage account, and packages should be pushed to the\u00a0dotnet5<\/code>\u00a0Azure DevOps NuGet feed. On the other hand, the channels used for the engineering services and tools push to the\u00a0dotnet-eng<\/code>\u00a0NuGet package feed. What this means is that the\u00a0intent<\/strong>\u00a0of a build does not have to be known at build time. Channel assignment indicates intent\u00a0and<\/em>\u00a0ensures that the outputs are published to the right locations based on that intent.<\/p>\nA little bit more on channels and intent<\/h4>\n
dotnet\/sdk<\/code>\u00a0repo pulls in a wide variety of input dependencies for .NET 5 Preview 8. One of those is the NuGet client. NuGet tends to produce new builds, test them, and then pick and choose which builds are ready for insertion into Visual Studio or the .NET SDK. Rather than have the\u00a0dotnet\/sdk<\/code>\u00a0team keep track of which branches NuGet is using at the time and the state of the builds, they simply set up a subscription to target the\u00a0release\/5.0.1xx-preview8<\/code>\u00a0with NuGet.Client builds assigned to the ‘VS 16.8’ channel. Similarly, the NuGet team also doesn’t need to anything about the SDK’s branching structure, only that they want builds intended for VS 16.8.<\/p>\nCoherency and Incoherency<\/h3>\n
Producing and Releasing a Product<\/h2>\n
\n
Code Flow<\/h3>\n
\n
release\/5.0<\/code>\u00a0branch in GitHub’s\u00a0dotnet\/runtime<\/code>\u00a0repo, there is a\u00a0release\/5.0<\/code>\u00a0branch in the Azure DevOps\u00a0dotnet-runtime<\/code>\u00a0repo and the\u00a0HEAD<\/code>\u00a0always matches. Every time a commit is made into GitHub, the\u00a0dotnet-runtime<\/code>\u00a0repo does a fast-forward merge to pull in the new changes.<\/li>\ninternal\/release\/5.0<\/code>\u00a0branch that is related to the\u00a0release\/5.0<\/code>\u00a0branch<\/strong>\u00a0– Every time a commit is made to GitHub’s\u00a0release\/5.0<\/code>\u00a0branch, the commit is\u00a0merged<\/em>\u00a0into\u00a0internal\/release\/5.0<\/code>. If there are no security fixes committed to the\u00a0internal\/release\/5.0<\/code>\u00a0branch, its HEAD matches the\u00a0release\/5.0<\/code>\u00a0branch. If there are, then\u00a0HEAD<\/code>\u00a0diverges.<\/li>\n<\/ul>\n\n
internal\/release\/...<\/code>\u00a0branches are unused. All commits and dependency flow occur against the GitHub\u00a0release\/...<\/code>\u00a0branches.<\/li>\ninternal\/release\/...<\/code>\u00a0branch in Azure DevOps and dependency flow targets the\u00a0internal\/release\/...<\/code>\u00a0branch in each repo. Any changes that are not security or dependency flow related are committed to the corresponding public GitHub branch, then automatically merged into the\u00a0internal\/release\/...<\/code>\u00a0branch. On release day, we open a PR to merge the\u00a0internal\/release\/...<\/code>\u00a0branch back into the public release branch (e.g. if commit was in\u00a0internal\/release\/5.0<\/code>, we merge it back to\u00a0release\/5.0<\/code>). This resets the internal branch state back to matching the public branch state, readying us for the next release.<\/li>\n<\/ul>\nShipping and Non-Shipping Assets<\/h3>\n
\n
https:\/\/nuget.org<\/code>, the .NET download sites, etc. These assets typically get “stable” versions (e.g. 5.0.6) rather than “non-stable” versions (e.g. 5.0.6-servicing.20364.11) when building for RTM and servicing releases.<\/li>\ndotnet\/sdk<\/code>\u00a0produces a package called “Microsoft.NET.Sdk” which is only redistributed from within the\u00a0dotnet\/installer<\/code>\u00a0output (zip and tar.gz files, for example) and does not appear on\u00a0https:\/\/nuget.org<\/code>. We always use non-stable versioning for non-shipping packages so that their versions remain unique build to build.<\/p>\nStable builds and Dependency Flow<\/h3>\n
Microsoft.Extensions.Logging<\/code>\u00a0is built out of\u00a0dotnet\/runtime<\/code>, has its contents redistributed in the\u00a0dotnet\/aspnetcore<\/code>\u00a0shared framework, but\u00a0also<\/em>\u00a0ships independently to\u00a0https:\/\/nuget.org<\/code>. When we build a new version of\u00a0Microsoft.Extensions.Logging<\/code>\u00a0for a release, we want it to have a stable\u00a0and<\/em>\u00a0predictable version number. We want\u00a05.0.6<\/code>, not\u00a05.0.6-servicing.20364.11<\/code>. But what happens when we need to build\u00a0dotnet\/runtime<\/code>\u00a0more than once for a given release? The first build of\u00a0dotnet\/runtime<\/code>\u00a0built\u00a0Microsoft.Extensions.Logging<\/code>\u00a0version\u00a05.0.6<\/code>, and the subsequent build also did the same. As a rule, NuGet feeds are immutable. We can’t simply overwrite the old package, nor would we want to. So how do we flow this new instance of\u00a0Microsoft.Extensions.Logging<\/code>\u00a0and ensure that the downstream repos pick up the correct package?<\/p>\n\n
\n
NuGet.config<\/code>\u00a0file and is publicly accessible.<\/li>\nNuGet.config<\/code>\u00a0of a repo when doing official builds. Adding it at build time avoids NuGet restore issues when building publicly, where it is not accessible. This feed only recieves new packages when doing internal builds.<\/li>\n<\/ul>\n<\/li>\n\n
NuGet.config<\/code>\u00a0file and is publicly accessible.<\/li>\nNuGet.config<\/code>\u00a0of a repo when doing official builds. Adding it at build time avoids NuGet restore issues when building publicly, where it is not accessible. This feed only recieves new packages when doing internal builds.<\/li>\n<\/ul>\n<\/li>\nNuGet.config<\/code>\u00a0file in a dependency update PR to include any feeds required to access the desired stable packages. Each build ensures a clean NuGet cache to avoid accidentally picking up the wrong instance of a stable package. On release day these assets are pushed to\u00a0https:\/\/nuget.org<\/code>\u00a0and any isolated feed containing packages with shipped versions are cleared out. This ensures that\u00a0https:\/\/nuget.org<\/code>\u00a0is the “source of truth”. Subsequent Maestro dependency updates will automatically remove these isolated feeds as they are no longer needed. A sample NuGet.config is shown below, from\u00a0https:\/\/github.com\/dotnet\/cli\/blob\/v3.1.300\/NuGet.config:<\/a><\/li>\n<\/ul>\n<?xml version=\"1.0\" encoding=\"utf-8\"?>\r\n<configuration>\r\n <packageSources>\r\n <clear \/>\r\n <!--Begin: Package sources managed by Dependency Flow automation. Do not edit the sources below.-->\r\n <add key=\"darc-int-dotnet-core-setup-0c2e69c\" value=\"https:\/\/pkgs.dev.azure.com\/dnceng\/_packaging\/darc-int-dotnet-core-setup-0c2e69ca\/nuget\/v3\/index.json\" \/>\r\n <add key=\"darc-int-dotnet-corefx-059a4a1\" value=\"https:\/\/pkgs.dev.azure.com\/dnceng\/_packaging\/darc-int-dotnet-corefx-059a4a19\/nuget\/v3\/index.json\" \/>\r\n <!--End: Package sources managed by Dependency Flow automation. Do not edit the sources above.-->\r\n <add key=\"dotnet-core\" value=\"https:\/\/dotnetfeed.blob.core.windows.net\/dotnet-core\/index.json\" \/>\r\n <add key=\"dotnet-tools\" value=\"https:\/\/pkgs.dev.azure.com\/dnceng\/public\/_packaging\/dotnet-tools\/nuget\/v3\/index.json\" \/>\r\n <add key=\"dotnet3\" value=\"https:\/\/pkgs.dev.azure.com\/dnceng\/public\/_packaging\/dotnet3\/nuget\/v3\/index.json\" \/>\r\n ...\r\n <\/packageSources>\r\n<\/configuration><\/pre>\n
https:\/\/nuget.org<\/code>\u00a0feed means that the NuGet restore behavior seen\u00a0<\/span>after<\/em>\u00a0a product releases will closely match how we validate pre-release. Non-shipping packages cannot accidentally appear in the restore graph. This also gives developers a cleaner\u00a0<\/span>NuGet.config<\/code>\u00a0file when consuming daily development builds. See\u00a0<\/span>Installers and Binaries<\/a>\u00a0and\u00a0<\/span>Sample NuGet Config<\/a>\u00a0examples.<\/span><\/p>\nInternal isolated feeds and public builds<\/h4>\n
NuGet.config<\/code>\u00a0file are not publicly available. While these feeds are empty after release day, simply their inclusion in the\u00a0NuGet.config<\/code>\u00a0file will cause the NuGet restore step to fail. This behavior has long been a complaint among the .NET teams. On release day, the internal to public merge PRs require manual fixups (removal of any darc-int- feeds) to pass. The good news is that this shouldn’t be a problem for future releases. The .NET Engineering Services team has recently merged a change into Maestro which disables the “darc-int” NuGet sources by default. Therefore, a public build of a repo will “assume” any internal source is not required (which is true after release day), while internal builds will automatically enable those sources. Going forward, this should ensure that commits tagged for release are always buildable without source modification on the day of release.<\/p>\nPrep and Validation<\/h3>\n
dotnet\/installer<\/code>, Maestro traverses the dependency graph starting at the\u00a0dotnet\/installer<\/code>\u00a0commit that we wish to ship and finds every unreleased dependency that contributes to that graph. It then locates all individual repo builds associated with those dependencies and the full set of Microsoft CI assets produced by those builds. Each of these assets has a set of endpoints associated with them, which is updated upon publishing. Maestro locates and downloads all the assets, placing them in a directory that is ready to be validated and released.<\/p>\n\n
Release!<\/h3>\n
https:\/\/dot.net<\/code>,\u00a0https:\/\/nuget.org<\/code>, docker registries, etc.) and publishing release notes and blog posts. GitHub repos are tagged at the commit that was built for release (e.g.\u00a0dotnet\/runtime preview 7 tag<\/a>), and if the release was built internally to address a security issue, the\u00a0internal\/release\/...<\/code>\u00a0branches are merged back into the corresponding public GitHub branches.<\/p>\n