NOTE: This blog was originally posted to the App Service Team blog
In this post, we get a behind-the-scenes look at the engineering work required to change a critical platform component with code paths that are exercised billions of times a day while minimizing service disruptions and maintaining SLA for our customers. We provide a brief introduction to help cover the basics, go over motivations for doing this work, explain some of the more interesting challenges, issues, and bugs encountered along the way, and close with the results and the new customers scenarios enabled.
The challenge was huge, but we’re excited about the benefits this brings to Azure App Services and our customers:
- Almost 80% improvement in throughput in performance tests designed to isolate the benefits.
- Greener Azure data centers from significantly decreased per-request CPU usage.
- Support for modern protocols like HTTP/3.
- Support for new customer scenarios such as gRPC applications, per-host cipher suite configuration, custom error pages, and more.
Introduction
In 2021, a group of engineers across multiple teams, including .NET and Azure, got together to transition the App Service Frontend fleet to Kestrel + YARP. As we celebrate the completion of this major lift and collaboration, we decided to write down the journey and describe some of the challenges of completing such a change to a live service, the wins we achieved, and the future work enabled by this transition. We hope you enjoy it.
Azure App Service in a nutshell
Azure App Service recently celebrated its 10 year anniversary (we launched it on June 7th, 2012). We are grateful and humbled by our customers who have helped us grow into a big service (affectionately called an XXL service in Azure internally, a designation only shared with 3 other services). Here are some numbers that provide a glimpse into our scale:
- 160B+ daily HTTP requests served by applications
- 14M+ host names
- 1.5K+ multi-tenant scale units and an additional 10K+ dedicated scale units (App Service Environments aka App Service Isolated SKU)
One of the key architectural pieces of this system is our FrontEndRole. The FrontEndRole main purposes are:
- Receiving traffic on HTTP/HTTPS from public virtual IP addresses associated with a scale unit
- Terminating SSL if required
- Determining which set of VMs are the origin-servers for the application (called Workers) and then routing to them
App Service was originally built as a Cloud Service and this role is just called FrontEndRole. With our transition to VM Scale Sets, the FrontEndRole is a separate scale set which is part of each scale unit.
The original App Service FrontEndRole, which runs on Windows Server, consisted of:
- IIS running on HTTP.sys, both operating system components of Windows Server
- Application Request Routing (ARR), which does request forwarding using WinHTTP
Kestrel and YARP in a nutshell
The first release of .NET Core introduced the Kestrel webserver: an open-source, cross-platform, and fast webserver implementation built using modern .NET. Performance is a key focus for the .NET team, and with each .NET release, Kestrel has gotten ever faster and more full-featured. As an example, recent changes made to Kestrel include:
- Significant scalability improvements on many-core machines
- Significant HTTP/2 performance enhancements when running with many concurrent streams
- Support for new standards like HTTP/3
YARP (“Yet Another Reverse Proxy”) is a reverse proxy toolkit that enables building fast proxy servers using infrastructure from ASP.NET and .NET, focusing on easy customization. It is developed in the open at https://github.com/microsoft/reverse-proxy. YARP’s toolkit/extensibility model made it easy for us to incorporate our routing and TLS handling with its request forwarding capabilities. YARP includes support for modern protocols like HTTP/2 & HTTP/3, which App Service customers can now expose. In addition, being based on the fast-evolving .NET platform means that every release, Kestrel and YARP benefit from improvements up and down the .NET stack, including everything from networking libraries all the way down to JIT compiler improvements that improve the quality of generated code. For a sampling of the types of improvements that went into just the .NET 6 release in 2021, see Performance Improvements in .NET 6.
Betting on Kestrel + YARP for App Service: Why?
The previous FrontEndRole architecture of App Service built on IIS/HTTP.SYS has served us well, but the promise of a modern HTTP stack in Kestrel + YARP could deliver new benefits to all App Service customers. Specifically:
- Performance improvements, including significantly decreased per-request CPU cost and per-connection memory cost.
- More flexible extensibility points into SSL termination path, allowing for easier dynamic SNI host selection.
- Enable new customer scenarios like support for gRPC, per-host cipher suite configuration, custom error pages, and more.
With all that context and motivation, the goal of the V-Team was clear:
“Transition the 200K+ dedicated cores running FrontEndRole to use Kestrel + YARP (and thus move away from IIS/HTTP.SYS/ARR)”
Challenge: Server Framework Diversity
App Service is not the first Microsoft service to transition to Kestrel and YARP. Microsoft has already documented the journeys of Bing, Azure Active Directory (AAD), and Dynamics 365 to .NET; these efforts have proven out the stability and performance of .NET for critical service workloads.
- Bing.com runs on .NET Core 2.1!
- Azure Active Directory’s gateway is on .NET 6.0!
- Dynamics 365 using YARP
The unique challenge that App Service adds to the mix is the diversity of server implementations. The previously mentioned Microsoft services are written by server engineers working for … Microsoft. This is definitely not the case for App Service, which enables customers to bring their own server frameworks and write their own applications with varying levels of standards compliance. Hosting customer applications brings a unique set of challenges described below.
Challenge: Platform versus Organic Health
Because App Service enables customers to write their own applications, the concept of “service health” is a nuanced discussion. App Service measures the health of the platform; we ensure that customers have a running VM which can connect to storage and can execute a simple canary request. But App Service cannot easily measure the organic health (HTTP request success rate) since we do not control the application. As a result, we primarily focused on platform health as our main metric.
For our transition to Kestrel + YARP, we needed to broaden our measurement to include organic health. Rather than looking for an absolute bar (say >99.99% success), we needed to compare “before Kestrel + YARP” and “after Kestrel + YARP” organic success and look for anomalies that would point out potential problems.
Challenge: Quick Rollback in Production
With a broadened approach to assessing organic health anomalies caused by a diverse set of applications/frameworks on our platform, we required fast mechanisms to undo our Kestrel + YARP transition on a per scale-unit basis; in other words, we needed to be able to “break glass” quickly when we encountered problems and return to using IIS/HTTP.SYS.
The Journey: 100% FrontEndRoles using Kestrel/YARP
With all the context and challenges described, here is how the journey looked like in a picture.
As you can see this journey took a lot of time. 6 months passed between the first Kestrel/YARP deployment and 100%.
The Bugs Encountered
We encountered multiple bugs on our journey to Kestrel + YARP. Apart from bugs in our business logic, one of the interesting classes of issue we encountered was the treatment of edge-cases in the HTTP specification. A diversity of clients hit our FrontEndRole. We need to be generous in accepting behavior that may not be exactly the HTTP spec’s letter and intent.
A simple example of one of these cases is when a request has leading newline characters (CR and/or LF). Strictly speaking, this isn’t allowed, but it turns out that there are some clients that send requests that start like:
\rGET / HTTP/1.1\r\n
...This is a case that IIS (and some other servers) allow, but because Kestrel historically has taken a fairly strict stance, we saw its parser rejecting requests like this with a BadHttpRequestException. Working closely with the ASP.NET Core team, we were able to make Kestrel a bit more generous in what it accepts (the example above now works in Kestrel in .NET 6.0.5 and newer releases).
Some other interesting issues uncovered can be found here, here and here.
As a result of investigating and addressing this class of issues, we’ve made Kestrel a more capable server without compromising the core principle of security.
The Payoff: Performance and New Features, Now and in the Future
Now that we have moved our FrontEndRoles to Kestrel + YARP we have realized multiple benefits in production.
Performance tests designed to isolate the benefits of our FrontEndRole change showed an almost 80% improvement in throughput (tested using a simple 1K helloworld response from a single dedicated worker in a test environment). App Service over-provisions FrontEndRole instances, so the realized benefit across our aggregate fleet is a large decrease in CPU% which provides more CPU headroom for the fleet. We are still in the early days of monitoring the fleet post-move; we may eventually be able to decrease our cores assigned to this role to reduce operating costs and data center energy usage. More investigation to follow.
With our move to Kestrel + YARP on our FrontEndRoles, we were also able to move our Linux worker VMs to use Kestrel+YARP. This change allows us to replace nginx, commonize the codebase, and light up gRPC for our App Service Linux SKUs. gRPC support has been a popular feature request from Azure App Services users and we’re excited to add this capability.
With this platform work complete we are now working on enabling two of the most frequently requested features in App Service; more news coming soon as we complete these improvements:
- Ability to configure custom error pages for requests that terminate on the front end (Specifically: HTTP 503, HTTP 502 and HTTP 403).
- Ability to specify TLS cipher suite allowed per given application. Today customers can only configure allowed cipher suites on our Isolated SKUs.
A Great Partnership
Once you have a live multi-tenant service running with millions of VMs globally, you learn to be very careful with how and when you advance it. That said, the innovations with Kestrel + YARP being developed by the .NET core team were just too valuable to pass up. At the same time, the .NET team would tell you the experience of supporting this migration was a whole new challenge for that team as they experienced the breadth and diversity of App Service scenarios. This was a great journey for both teams and we landed it. Now that we have this new platform in place, we look forward to continued innovation between our teams.
What I need to do if the migration was done?
For eg: is the web.config still work?
There is no action required. This change is not done on the webserver where your app is hosted, but rather on common infrastructure that acts as a load balancer and should be 100% transparent to you.
Does that mean we will have HTTP 3 support soon?
It’s unblocked, can’t comment on timeline at the moment but this was the first step towards unblocking that scenario.
Wow, great work! That’s a daunting task, making sure all the existing sites still work as expected! You listed three interestingissues, how many issues in total did your organic health checks find? And did you see much improvement in the memory footprint vs IIS?
Thanks Byron for the post. Quite interesting to see Microsoft still does some dogfooding 😏
Yet you forgot to mention one thing: which is the host operating system for the new Frontend role built upon Kestrel + YARP? I hope that still is Windows… 🤔🤔
Another very interesting news is that you replaced Nginx with Kestrel. Awesome.
We are using Windows for the App Service Front Ends, but I think that is beside the point. The more important thing in my opinion is that if you are writing a .NET app you can rely on the same battle tested components that we use to run global scale service and the same code can run anywhere from a RaspberryPI to a Mac Laptop to a Windows Server or a planet scale PaaS offering.
Byron,
I definitely agree. From my LinkedIn post about your post:
The juicy news here is not only that Kestrel + YARP is being used at massive scale by MS itself, quite a proof, but also that Azure is replacing Nginx with Kestrel + YARP for Linux worker VMs. Which is indeed interesting.
However, my question had a goal. If you at Microsoft are gradually but steadily replacing Windows Server with Linux, WS IS the technology we shouldn't...
There are places like WAF v2 (nginix under the covers) where we switched to Linux because it made more sense to build off something that is super common across the industry vs. building from scratch. On the other hand our deep experience with Windows virtualization means that all of our physical hosts underneath Azure VMs are Windows, as are the VMs underneath our various flavors of SQL Server. Then, we have solutions like Arc or...
See what I mean?
https://azure.github.io/AppService/2022/08/11/gRPC-support-on-App-Service-now-in-Public-Preview.html
Not even mentioned when it will be available for Windows. Now I don’t care about the service itself but I got the message.
Evan, I understand this explanation under the service provider point of view and as a service provider, we tell our customers the same thing: we provide to you anything, use what you want.
However, when you're building a solution or developing a solution it is not that you would use tens of Web servers or tens of other components that do the same thing. You usually pick a (very) short list of them, possibly one of...
It is still running on windows yes.
Thank you, David. That’s what I hoped.