{"id":15765,"date":"2017-12-27T22:31:45","date_gmt":"2017-12-27T22:31:45","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/vsoservice\/?p=15765"},"modified":"2019-02-18T13:42:58","modified_gmt":"2019-02-18T21:42:58","slug":"postmortem-intermittent-failures-for-visual-studio-team-services-on-14-dec-2017","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/devopsservice\/postmortem-intermittent-failures-for-visual-studio-team-services-on-14-dec-2017\/","title":{"rendered":"Postmortem – Intermittent Failures for Visual Studio Team Services on 14 Dec 2017"},"content":{"rendered":"

On <\/span>14 <\/span>December <\/span>2017 <\/span>we <\/span>began to have <\/span>a series of incidents with Visual Studio Team Services (VSTS) <\/span>for several days <\/span>that had a serious impact on the availability of our service for many customers (incident blogs <\/span>#1<\/span><\/a> #2<\/span><\/a> #3<\/span><\/a>). We apologize for the disruption<\/span> these incidents had on you and your team<\/span>. Below we describe the cause and the actions we are taking to address the issues<\/span> which caused these incidents<\/span>.<\/span>\u00a0<\/span><\/p>\n

Customer Impact<\/strong><\/p>\n

The issues <\/span>caused intermittent failures across multiple instances of the VSTS service within <\/span>certain<\/span> US and Brazil<\/span>ian <\/span>data centers<\/span>. During this time<\/span>,<\/span> we experienced failures within our application which caused IIS <\/span>restarts <\/span>resulting in customer impact for various VSTS scenarios.\u00a0<\/span>\u00a0<\/span><\/p>\n

The incident<\/span>s<\/span> started on 14 December. The graph below shows periods of customer impact for the Central US (CUS) and South Brazil (SBR) scale units.<\/span>\u00a0<\/span><\/p>\n

\"\"<\/a><\/p>\n

What Happened<\/strong><\/p>\n

For context VSTS uses DNS <\/span>to route traffic<\/span> to the correct scale unit. <\/span>When <\/span>accounts are created, <\/span>VSTS queues a job to create a DNS record for {account<\/span>}.visualstudio.com<\/span> which <\/span>point<\/span>s<\/span> to the right scale unit. Because there is a delay between the DNS entry being added and <\/span>it <\/span>being used, we use <\/span>Application Request Routing<\/span><\/a> (ARR) to re-route requests to the right scale unit until the DNS update is <\/span>complete<\/span>. Additionally, VSTS uses web sockets to provide real time updates in the browser for pull requests and builds via <\/span>SignalR<\/span>.\u00a0<\/span>\u00a0<\/span><\/p>\n

The application pools (w3wp process) <\/span>for <\/span>the Brazil and Central US VSTS scale units began crashing intermittently on <\/span>14 <\/span>December<\/span>. IIS <\/span>restart<\/span>ed<\/span> the application pools on failure, but all existing connections <\/span>were <\/span>terminated. Analysis of the dumps revealed that a certain call pattern <\/span>trigger<\/span>ed<\/span> the crash.\u00a0<\/span>\u00a0<\/span><\/p>\n

The request characteristics common to each crash were the following.<\/span>\u00a0<\/span><\/p>\n

    \n
  1. The request was a web socket request.<\/span>\u00a0<\/span><\/li>\n<\/ol>\n
      \n
    1. The request was proxied using ARR.<\/span>\u00a0<\/span><\/li>\n<\/ol>\n

      The issue took a while to track down because we suspected recent changes to the code that uses <\/span>SignalR<\/span>. However, the root cause was that on <\/span>14 <\/span>December <\/span>we <\/span>released a fix to an unrelated <\/span>issue<\/span> which <\/span>added code to use the ASP.Net <\/span>PreSendRequestHeaders<\/span><\/a> event.\u00a0 Using this event in combination with web sockets and ARR caused an <\/span>AccessViolationException<\/span> which <\/span>terminat<\/span>ed<\/span> the process. We spoke with the ASP.Net team and they informed us that the <\/span>PreSendRequestHeaders<\/span> method is unreliable and we should replace it with <\/span>HttpResponse.AddOnSendingHeaders<\/span><\/a> instead. We have released a fix with that change.<\/span>\u00a0<\/span><\/p>\n

      While debugging the issue, we mitigated customer impact by redirecting ARR traffic once we realized that was the key cause.<\/span>\u00a0<\/span><\/p>\n

      Here is a timeline of the incident.<\/span>\u00a0<\/span><\/p>\n

      \"\"<\/a><\/p>\n

        \n
      1. IIS errors start in SBR<\/span><\/span> (Brazil South)<\/span><\/span>\u00a0<\/span><\/li>\n
      2. IIS errors start in CUS1<\/span> (<\/span>US Central)<\/span>\u00a0<\/span><\/li>\n
      3. Workaround \u2013 stopped ARR traffic from going to SBR.<\/span>\u00a0<\/span><\/li>\n
      4. Workaround \u2013 Redirected *.visualstudio.com wildcard from CUS1 to pre-flight (an internal instance).\u00a0<\/span><\/li>\n<\/ol>\n

        Next Steps<\/strong><\/p>\n

        In order to prevent this issue in the future, we are taking the following actions.<\/p>\n

          \n
        1. We have added monitoring and alerting specifically for w3wp crashes.<\/li>\n
        2. We are working with the ASP.NET team to document or deprecate the PreSendRequestHeaders method. This page<\/a> has been updated, and we are working to get the others updated.<\/li>\n
        3. We are adding more detailed markers to our telemetry to make it easier to identify which build a given scale unit is on at any point in time to help correlate errors with the builds that introduced them.<\/li>\n<\/ol>\n

          Sincerely,\nBuck Hodges\nDirector of Engineering, VSTS<\/p>\n","protected":false},"excerpt":{"rendered":"

          On 14 December 2017 we began to have a series of incidents with Visual Studio Team Services (VSTS) for several days that had a serious impact on the availability of our service for many customers (incident blogs #1 #2 #3). We apologize for the disruption these incidents had on you and your team. Below we […]<\/p>\n","protected":false},"author":1090,"featured_media":18098,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[2,1],"tags":[3],"class_list":["post-15765","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-postmortem","category-uncategorized","tag-postmortem"],"acf":[],"blog_post_summary":"

          On 14 December 2017 we began to have a series of incidents with Visual Studio Team Services (VSTS) for several days that had a serious impact on the availability of our service for many customers (incident blogs #1 #2 #3). We apologize for the disruption these incidents had on you and your team. Below we […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/devopsservice\/wp-json\/wp\/v2\/posts\/15765","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/devopsservice\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/devopsservice\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devopsservice\/wp-json\/wp\/v2\/users\/1090"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devopsservice\/wp-json\/wp\/v2\/comments?post=15765"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/devopsservice\/wp-json\/wp\/v2\/posts\/15765\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devopsservice\/wp-json\/wp\/v2\/media\/18098"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/devopsservice\/wp-json\/wp\/v2\/media?parent=15765"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devopsservice\/wp-json\/wp\/v2\/categories?post=15765"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devopsservice\/wp-json\/wp\/v2\/tags?post=15765"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}