We’ve blogged previously about how the Azure SDK team does packaging and tooling. We also shared some of the thinking behind our mono-repo strategy. However, it was recently pointed out to us that we really haven’t shared much about how our release process works. So let’s pull back the curtain on how the Azure SDK team uses GitHub, Azure DevOps, and various other tools and policies to deliver SDK libraries to developers.
I’ve been on the centralized Azure SDK team since it was formed nearly four years ago. Because I like to tell stories, I’m going to walk you through some history as I share what we do and how we do it.
Look ma, I’m quadrilingual (or octolingual?!)
First, let’s talk about the problem landscape. When the team was formed, it was decided that we would deliver high quality SDKs that would adhere to general and language-specific guidelines. The languages we initially targeted were:
Producing SDKs in these four languages meant that the engineering system would need to support building packages and releasing them to four distinct package feeds: NuGet, Maven, PyPi, and npm respectively. Our release pipelines would have to support four different toolsets to be able to release to those feeds. Two of those feeds, NuGet and Maven, support package signing, so we’d have to support that as well for those languages.
More recently, we’ve expanded our language support to several other languages and ecosystems including C, C++, Go, Android, and iOS. Each of these languages has its own unique tooling, release processes, and feeds.
Setting Boundaries
One of the earliest decisions we made regarding our engineering system was security related. At the time, GitHub actions barely existed, and Microsoft was still investing heavily in Azure DevOps. So, we created our own Azure DevOps organization for this new engineering system.
We created two key projects in our Azure DevOps organization: “public” and “internal”. All pipelines would be run in one of these two projects.
The “public” project would be where all PR validation pipelines would run. It would only run recording based tests (no tests against live services) and would have no permissions or secrets for accessing our Azure subscriptions, the corporate network, or any other sensitive resources. This separation was important because the Azure SDKs are all open source and maintained in our public repos. It is our goal to foster an open source community and we invite contributions and PRs from anyone at any time. Therefore, our PR validation needed to be locked down so that someone couldn’t submit a malicious PR that tried to use our resources for inappropriate purposes.
Our Azure DevOps “internal” project is where live tests run. It’s also where we run our release pipelines, which publish our SDK packages to the various public feeds I mentioned above. Pipelines running in the “internal” project have access to our subscriptions so that they can create, use, and clean up resources for live testing. These pipelines also have access to our service account credentials, which can publish to the various package feeds. Only members of the Azure SDK team can run pipelines in the “internal” Azure DevOps project.
Pipeline Evolution
As we started down the path of defining guidelines for how to make excellent SDK libraries and began implementing the first of them, our original pipeline strategy was simple. Hand-written pipeline yaml configs were checked into our repos for each language. We had one pipeline per language that built all of our new libraries following the general and language-specific guidelines. For each language, we had a pipeline for PR validation, which ran in “public,” and another pipeline for live testing and publishing releases, which ran in “internal”.
Later, we moved to our current pipeline model, which is organized around services. The repo structure for each of our language mono-repos is based on services. Each service has its own folder where all of the SDK code, tests, samples, readmes, etc. for that service live (Example: Storage in JavaScript). The second iteration of our pipeline strategy was based off of this breakdown. Each service has its own set of pipelines for PR validation, live tests, and release that will build/test/release the SDK code in their service folder. When a PR is submitted, we only run PR validation pipelines for the service folders where changes have been made. Each service has a live test pipeline that runs daily off of the main branch and verifies the libraries against live service endpoints. When a team member needs to release SDK libraries, they run the release pipeline for that service folder. The release pipeline will allow them to release some or all of the SDK libraries for that service.
We have pipeline generation tools to ease the “busy work” associated with this system. Whenever a new service folder and pipeline configuration are added, our generator tool automatically creates the full set of pipelines. We also regularly run the pipeline generator to ensure consistency across all of our pipelines.
Tag, you’re “it”
We generally don’t do release branches. In the Azure SDK repos, the main branch is intended to be a live view of our actively developed libraries. The main branch should always be buildable, but often has yet-to-be-released features and bug fixes in it. Once a month, we release any libraries with pending changes. With each release of a library, we create a tag in the repo pointing to the SHA it was released under. The tag is created so that later, if that package needs a hotfix, a branch can be created off of the same SHA used to release it. Then the hotfix can be implemented in the branch, released (which creates a new release tag), and merged back to main, if necessary. Then the hotfix branch can be deleted.
By creating git tags with each release, we significantly reduce the need for long-lived branches in our repos. If you want to read more about our branching strategies and guidelines, we have great documentation.
Always have the right tools for the job
All common scripts used in our engineering system across languages are maintained in the azure-sdk-tools repo. You’ll find language-specific copies of these tools in the /eng/common folder of each of the language repos (Example: Java). Because we have an automatic process to mirror these shared scripts into the language repos, we don’t ever change them in the /eng/common folder of the language repos. If we did, they would be overwritten by this automatic mirror.
Additionally, each language has its own /eng directory that contains language-specific build/test/release templates. These templates utilize the common building blocks I mentioned above along with tools and compilers as required by each language.
The reason we have this process is to ensure the tools always match the code. You may need to sync to an old SHA and build an older library version based off a tag. The matching tools and scripts for building those libraries are found in the /eng/common folder at the same SHA. While this system isn’t 100% effective (because things like build host OS images move for example), it gets us a long way towards having “temporally reliable” pipelines.
Conclusion
So I’ve covered, at a high level, much of what we do to release SDK libraries here. My hope is that this blog post opens the door for questions to be asked and more blog posts to be created answering them. Let us know in the comments if you have specific questions or would like us to go deeper in any of these areas.
Lastly, I just want to say how much I enjoy working with the folks on the Azure SDK team. I work with amazing people who sincerely want to help each other do their best work. We’re creating tools and SDKs that are critically important for developers building on top of Azure. I truly work on the greatest team in the world!
0 comments