{"id":37367,"date":"2021-11-15T08:00:11","date_gmt":"2021-11-15T15:00:11","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/dotnet\/?p=37367"},"modified":"2022-05-03T20:59:56","modified_gmt":"2022-05-04T03:59:56","slug":"azure-active-directorys-gateway-is-on-net-6-0","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/dotnet\/azure-active-directorys-gateway-is-on-net-6-0\/","title":{"rendered":"Azure Active Directory&#8217;s gateway is on .NET 6.0!"},"content":{"rendered":"<p>Azure Active Directory\u2019s gateway service is a reverse proxy that fronts hundreds\nof services that make up Azure Active Directory (Azure AD). If you\u2019ve used\nservices such as office.com, outlook.com, portal.azure.com or xbox.live.com,\nthen you\u2019ve used Azure AD\u2019s gateway. The gateway provides features such as TLS\ntermination, automatic failovers\/retries, geo-proximity routing, throttling, and\ntarpitting to services in Azure AD. The gateway is present in 54 Azure\ndatacenters worldwide and serves <strong>~185 Billion<\/strong> requests each day. Up until\nrecently, Azure AD\u2019s gateway was running on .NET 5.0. As of September 2021, it\u2019s\nrunning on .NET 6.0.<\/p>\n<h2>Efficiency gains by moving to .NET 6.0<\/h2>\n<p>The below image shows that application CPU utilization dropped by <strong>33%<\/strong> for\nthe same traffic volume after moving to .NET 6.0 on our production fleet.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/dotnet\/wp-content\/uploads\/sites\/10\/2021\/11\/efficiency.png\" alt=\"efficiency\" \/><\/p>\n<p>The above meant that our application efficiency went up by <strong>50%<\/strong>. Application\nefficiency is one of the key metrics we use to measure performance and is\ndefined as<\/p>\n<pre><code class=\"language-xml\">Application efficiency = (Requests per second) \/ (CPU utilization of application)<\/code><\/pre>\n<h2>Changes made in .NET 6.0 upgrade<\/h2>\n<p>Along with the .NET 6.0 upgrade, we made two major changes:<\/p>\n<ol>\n<li>Migrated from IIS to <a href=\"https:\/\/docs.microsoft.com\/aspnet\/core\/fundamentals\/servers\/httpsys\">HTTP.sys\nserver<\/a>.\nThis was made possible by new features in .NET 6.0.<\/li>\n<li>Enabled dynamic\n<a href=\"https:\/\/devblogs.microsoft.com\/dotnet\/conversation-about-pgo\/\">PGO<\/a>\n(profile-guided optimization). This is a new feature of .NET 6.0.<\/li>\n<\/ol>\n<p>The following sections will describe each of those changes in more detail.<\/p>\n<h2>Migrating from IIS to HTTP.sys server<\/h2>\n<p>There are 3 server options to pick from in ASP.NET Core:<\/p>\n<ul>\n<li>Kestrel<\/li>\n<li>HTTP.sys server<\/li>\n<li>IIS<\/li>\n<\/ul>\n<p>A previous <a href=\"https:\/\/devblogs.microsoft.com\/dotnet\/azure-active-directorys-gateway-service-is-on-net-core-3-1\">blog\npost<\/a>\ndescribes why Azure AD gateway chose IIS as the server to run on during our .NET\nFramework 4.6.2 to .NET Core 3.1 migration. During the .NET 6.0 upgrade, we\nmigrated from IIS to HTTP.sys server. Kestrel was not chosen due to the lack of\ncertain <a href=\"https:\/\/github.com\/dotnet\/runtime\/issues\/45456\">TLS<\/a>\n<a href=\"https:\/\/github.com\/dotnet\/runtime\/issues\/60949\">features<\/a> our service depends\non (support is expected by June 2022 in Windows Server 2022).<\/p>\n<p>By migrating from IIS to HTTP.sys server, Azure AD gateway saw the following\nbenefits:<\/p>\n<ul>\n<li>A <strong>27%<\/strong> increase in application efficiency.<\/li>\n<li><strong>Deterministic queuing model:<\/strong> HTTP.sys server runs on a single-queue system,\nwhereas IIS has an internal queue on top of the HTTP.sys queue. The\ndouble-queue system in IIS results in unique performance problems (especially\nin high concurrency situations, although issues in IIS can potentially be\noffset by tweaking Windows registry keys such as <code style=\"font-size: 15px;\">HKLM:SYSTEM\\CurrentControlSet\\Services\\W3SVC\\Performance\\ReceiveRequestsPending<\/code>). By removing IIS and moving to a single-queue system\non HTTP.sys, queuing issues that arose due to rate mismatches in the\ndouble-queue system disappeared as we moved to a deterministic model.<\/li>\n<li><strong>Improved deployment and autoscale experience:<\/strong> The move away from IIS\nsimplifies deployment since we no longer need to install\/configure IIS and\n<a href=\"https:\/\/docs.microsoft.com\/aspnet\/core\/host-and-deploy\/aspnet-core-module\">ANCM<\/a>\nbefore starting the website. Additionally, TLS configuration is easier and\nmore resilient as it needs to be specified at just one layer (HTTP.sys)\ninstead of two as it had been with IIS.<\/li>\n<\/ul>\n<p>The following showcase some of the changes that were made while moving from IIS\nto HTTP.sys server:<\/p>\n<ul>\n<li><strong>TLS renegotiation:<\/strong> Renegotiation provides the ability to do optional client certificate negotiation\nbased on HTTP constructs such as request path.<\/li>\n<\/ul>\n<p style=\"padding-left: 40px;\">Example: On IIS, during the initial TLS handshake with the client, the server\ncan be configured to not request a client certificate. However, if the path\nof the request contains, say &#8220;foo&#8221;, IIS triggers a TLS renegotiation and\nrequests a client certificate.<\/p>\n<p style=\"padding-left: 40px;\">The following web.config configuration in IIS is how path based TLS\nrenegotiation is enabled on IIS:<\/p>\n<pre style=\"margin-left: 40px;\"><code class=\"language-xml\">  &lt;location path=\"foo\"&gt;\r\n      &lt;system.webServer&gt;\r\n        &lt;security&gt;\r\n          &lt;access sslFlags=\"Ssl, SslNegotiateCert, SslRequireCert\"\/&gt;\r\n        &lt;\/security&gt;\r\n      &lt;\/system.webServer&gt;\r\n  &lt;\/location&gt;<\/code><\/pre>\n<p style=\"padding-left: 40px;\">In HTTP.sys server hosting (.NET 6.0 and up), the above configuration is\nexpressed in code by calling\n<a href=\"https:\/\/docs.microsoft.com\/dotnet\/api\/microsoft.aspnetcore.http.connectioninfo.getclientcertificateasync\">GetClientCertificateAsync()<\/a>\nas below.<\/p>\n<pre style=\"margin-left: 40px;\"><code class=\"language-csharp\">  \/\/ default renegotiate timeout in http.sys is 120 seconds.\r\n  const int RenegotiateTimeOutInMilliseconds = 120000;\r\n  X509Certificate2 cert = null;\r\n  if (httpContext.Request.Path.StartsWithSegments(\"foo\"))\r\n  {\r\n    if (httpContext.Connection.ClientCertificate == null)\r\n    {\r\n      using (var ct = new CancellationTokenSource(RenegotiateTimeOutInMilliseconds))\r\n      {\r\n        cert = await context.Connection.GetClientCertificateAsync(ct.Token);\r\n      }\r\n    }\r\n  }<\/code><\/pre>\n<p style=\"margin-left: 40px;\">In order for <code class=\"language-charp\">GetClientCertificateAsync()<\/code> to trigger a renegotiation, the\nfollowing setting should be set in\n<a href=\"https:\/\/docs.microsoft.com\/dotnet\/api\/microsoft.aspnetcore.server.httpsys.httpsysoptions\">HttpSysOptions<\/a><\/p>\n<pre style=\"margin-left: 40px;\"><code class=\"language-csharp\">options.ClientCertificateMethod = ClientCertificateMethod.AllowRenegotation;<\/code><\/pre>\n<ul>\n<li><strong>Mapping IIS Server variables:<\/strong>\n<p style=\"margin-left: 40px;\">\n<\/li>\n<ul>\n<li>On IIS, TLS information such as <code>CRYPT_PROTOCOL<\/code>, <code>CRYPT_CIPHER_ALG_ID<\/code>,\n<code>CRYPT_KEYEXCHANGE_ALG_ID<\/code> and <code>CRYPT_HASH_ALG_ID<\/code> is obtained by <a href=\"https:\/\/docs.microsoft.com\/iis\/web-dev-reference\/server-variables\">IIS Server\nvariables<\/a>\nand can be leveraged as shown\n<a href=\"https:\/\/www.microsoft.com\/security\/blog\/2017\/09\/07\/new-iis-functionality-to-help-identify-weak-tls-usage\/\">here<\/a>.<\/li>\n<li>On HTTP.sys server, equivalent information is exposed via\n<a href=\"https:\/\/docs.microsoft.com\/dotnet\/api\/microsoft.aspnetcore.connections.features.itlshandshakefeature\">ITlsHandshakeFeature&#8217;s<\/a>\n<code>Protocol<\/code>, <code>CipherAlgorithm<\/code>, <code>KeyExchangeAlgorithm<\/code> and <code>HashAlgorithm<\/code>\nrespectively.<\/li>\n<\/ul>\n<li style=\"margin-top: 10px;\"><strong>Ability to interpret non-ASCII headers: <\/strong><\/li>\n<\/ul>\n<p style=\"margin-left: 40px;\">The gateway receives millions of headers each day with non-ASCII characters in them and the ability to interpret non-ASCII headers is important. Kestrel and IIS already have this ability, and in .NET 6.0, Latin1 request header encoding <a href=\"https:\/\/github.com\/dotnet\/aspnetcore\/issues\/34330\">was added for HTTP.sys<\/a> as well. It can be enabled using <code>HttpSysOptions<\/code> as shown below.<\/p>\n<pre style=\"margin-left: 40px;\"><code class=\"language-csharp\">options.UseLatin1RequestHeaders = true;<\/code><\/pre>\n<ul>\n<li><strong>Observability:<\/strong><\/li>\n<\/ul>\n<p style=\"padding-left: 40px;\">In addition to <a href=\"https:\/\/docs.microsoft.com\/dotnet\/core\/diagnostics\/available-counters\">.NET\ntelemetry<\/a>,\nthe health of a service can be monitored by plugging into a wealth of\n<a href=\"https:\/\/docs.microsoft.com\/windows\/win32\/http\/scenario-3--performance-counters\">telemetry<\/a>\nexposed by HTTP.sys such as:<\/p>\n<pre class=\"prettyprint\" style=\"margin-left: 40px;\">Http Service Request Queues\\ArrivalRate\r\nHttp Service Request Queues\\RejectedRequests\r\nHttp Service Request Queues\\CurrentQueueSize\r\nHttp Service Request Queues\\MaxQueueItemAge\r\nHttp Service Url Groups\\ConnectionAttempts\r\nHttp Service Url Groups\\CurrentConnections<\/pre>\n<h2>Enabling Dynamic PGO (profile-guided optimization)<\/h2>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/dotnet\/announcing-net-6-release-candidate-1\/#dynamic-pgo\">Dynamic\nPGO<\/a>\nis one the most exciting features of .NET 6.0! PGO can benefit .NET 6.0\napplications by maximizing <em>steady-state<\/em> performance.<\/p>\n<p>Dynamic PGO is an opt-in feature in .NET 6.0. There are 3 environment variables\nyou need to set to enable dynamic PGO:<\/p>\n<ul>\n<li><code>set DOTNET_TieredPGO=1<\/code>. This setting leverages the initial Tier0 compilation of\nmethods to observe method behavior. When methods are rejitted at Tier1, the\ninformation gathered from the Tier0 executions is used to optimize the Tier1\ncode. Enabling this switch increased our application efficiency by <strong>8.18%<\/strong>\ncompared to plain .NET 6.0.<\/li>\n<li><code>set DOTNET_TC_QuickJitForLoops=1<\/code>. This setting enables tiering for methods\nthat contain loops. Enabling this switch (in conjunction with above switch)\nincreased our application efficiency by <strong>10.2%<\/strong> compared to plain .NET 6.0.<\/li>\n<li><code>set DOTNET_ReadyToRun=0<\/code>. The core libraries that ship with .NET come with\nReadyToRun enabled by default. ReadyToRun allows for faster startup because\nthere is less to JIT compile, but this also means code in ReadyToRun images\ndoesn&#8217;t go through the Tier0 profiling process which enables dynamic PGO. By\ndisabling ReadyToRun, the .NET libraries also participate in the dynamic PGO\nprocess. Setting this switch (in conjunction with the two above) increased\nour application efficiency by <strong>13.23%<\/strong> compared to plain .NET 6.0.<\/li>\n<\/ul>\n<h2>Learnings<\/h2>\n<ul>\n<li>There were a few <code>SocketsHttpHandler<\/code> changes in .NET 6.0 that surfaced as\nissues in our service. We worked with the .NET team to identify workarounds\nand improvements.<\/p>\n<ul>\n<li>New connection attempts that fail <a href=\"https:\/\/github.com\/dotnet\/runtime\/issues\/60654\">can impact multiple\nHTTP requests<\/a> in .NET 6.0,\nwhereas a failed connection attempt would only impact a single HTTP\nrequest in .NET 5.0.<\/p>\n<ul>\n<li style=\"margin-bottom: 10px;\">Workaround : Setting a\n<a href=\"https:\/\/docs.microsoft.com\/dotnet\/api\/system.net.http.socketshttphandler.connecttimeout\">ConnectTimeout<\/a>\nslightly lower than HTTP request timeout ensures .NET 5.0 behavior is\nmaintained. Alternatively, disposing the underlying handler on a\nfailure also ensures only a single request is impacted due to a\nconnect timeout (although this can be expensive depending on the size\nof the connection pool, please be sure to measure for your scenario).<\/li>\n<\/ul>\n<\/li>\n<li>Requests that fail due to RST packets are <a href=\"https:\/\/github.com\/dotnet\/runtime\/issues\/60644\">no longer automatically\nretried<\/a> in .NET 6.0 and\nthis results in an elevated rate of <code>An existing connection was forcibly closed by the remote host<\/code> exceptions bubbling up to the application from\nHttpClient.<\/p>\n<ul>\n<li style=\"margin-bottom: 10px;\">Workaround : The application can add retries on top of HttpClient for\nidempotent requests. Additionally, if RST packets are due to idle\ntimeouts, setting\n<a href=\"https:\/\/docs.microsoft.com\/dotnet\/api\/system.net.http.socketshttphandler.pooledconnectionidletimeout\">PooledConnectionIdleTimeout<\/a>\nto lower than the idle timeout of the server will help eliminate RST\npackets due to idle connections.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li><code>HttpContext.RequestAborted.IsCancellationRequested<\/code> had inconsistent behavior\non HTTP.sys compared to other servers and has been\n<a href=\"https:\/\/github.com\/dotnet\/aspnetcore\/issues\/35407\">fixed<\/a> in .NET 6.0.<\/li>\n<li>Client side disconnects were <a href=\"https:\/\/github.com\/dotnet\/aspnetcore\/issues\/35490\">noisy on HTTP.sys\nserver<\/a> and there was a\n<a href=\"https:\/\/github.com\/dotnet\/aspnetcore\/pull\/35401\">race condition<\/a> that was\ntriggered while trying to set StatusCode on a disconnected request. Both have\nbeen fixed in .NET 6.0.<\/li>\n<\/ul>\n<h2>Summary<\/h2>\n<p>Every new release of .NET has <a href=\"https:\/\/devblogs.microsoft.com\/dotnet\/performance-improvements-in-net-6\/\">tremendous performance\nimprovements<\/a>\nand there is a huge upside to migrating to the latest version of .NET. For Azure\nAD gateway, we look forward to trying out newer APIs specific to .NET 6.0 for\neven bigger wins and further enhancements in .NET 7.0.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Read about gains from Azure Active Directory gateway service&#8217;s migration from .NET 5.0 to .NET 6.0.<\/p>\n","protected":false},"author":47836,"featured_media":37368,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[685,196,197,7509,327,7635],"tags":[],"class_list":["post-37367","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dotnet","category-dotnet-core","category-aspnet","category-aspnetcore","category-azure","category-developer-stories"],"acf":[],"blog_post_summary":"<p>Read about gains from Azure Active Directory gateway service&#8217;s migration from .NET 5.0 to .NET 6.0.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/37367","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/users\/47836"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/comments?post=37367"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/37367\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media\/37368"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media?parent=37367"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/categories?post=37367"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/tags?post=37367"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}