{"id":273,"date":"2014-11-19T06:32:23","date_gmt":"2014-11-19T06:32:23","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/bharry\/2014\/11\/19\/incident-response\/"},"modified":"2024-03-13T11:36:22","modified_gmt":"2024-03-13T18:36:22","slug":"incident-response","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/bharry\/incident-response\/","title":{"rendered":"Incident response"},"content":{"rendered":"

We had an incident on VS Online yesterday, starting at 1 AM UTC (5:00 PM PST).\u00a0 The incident was caused by an underlying Azure storage incident affecting many stamps in several regions.\u00a0 I don\u2019t know the root cause of that yet but I\u2019m sure I\u2019ll find out.<\/p>\n

I want to start by apologizing for the incident.\u00a0 No incident is good or acceptable.\u00a0 I know it disrupted important work happening for our customers and we take that very seriously.<\/p>\n

At the same time, they happen.\u00a0 Nothing you can do can ever 100% prevent them.\u00a0 As a result, how you respond to them is as important as the work you do to prevent them.\u00a0 I wanted to share with you all a mail I sent to my team this morning that, I think, captures both how well the team did and how I think about what\u2019s important.<\/p>\n

——- Begin mail thread ——–<\/p>\n

I want to congratulate the team for the hard work and great response to this incident.\u00a0 The availability graph below roughly corresponds to what Buck describes.\u00a0 It would be interesting to see the Azure storage availability graph overlaid so we could measure how much effect our own mitigations had over and above Azure\u2019s.<\/p>\n

Things you should be particularly proud of:<\/p>\n

    \n
  1. Detection<\/strong> \u2013 The issue was detected promptly (before customers reported it)<\/li>\n
  2. Response time<\/strong> \u2013 The team was on the issue within minutes.<\/li>\n
  3. Communication<\/strong> \u2013 Buck notified me about the incident at 5:21pm PST \u2013 that\u2019s within about 20 minutes of incident detection and within 10 minutes of the bridge beginning.\u00a0 More importantly, when I went to our support page<\/a> a few minutes later, I saw that we had public communication about incident awareness already up for 15 minutes.<\/li>\n
  4. Mitigation<\/strong> \u2013 I love the proactive steps taken below to figure out how to adjust the service to best mitigate the effects of the underlying outage.<\/li>\n
  5. Proactiveness<\/strong> \u2013 I love that we are going to look at how to make the service able to make those kinds of configuration changes to mitigate effect automatically in the future.<\/li>\n
  6. Telemetry<\/strong> \u2013 It\u2019s awesome that I can easily produce the graph in the mail below that very clearly shows the magnitude and duration of the incident (note there\u2019s more data on availability, etc that I didn\u2019t include in the picture below).<\/li>\n
  7. Tracking<\/strong> \u2013 There was great tracking of the exact incident and response.\u00a0 The fact that I can write this mail within 12 hours documenting what happened down to the minute is kind of amazing.\u00a0 It\u2019s hard to understate how difficult it is to avoid the \u201cfog of war\u201d that obscures what\u2019s actually going on in cases like this.\u00a0 Without a clear record of what happened and when, it\u2019s hard to be precise about where you can improve.<\/li>\n<\/ol>\n

    There\u2019s a lot here to be proud of.\u00a0 Great work all of you\u2026<\/p>\n

    \"Image<\/a><\/p>\n

    Brian<\/p>\n

    <\/a><\/p>\n

    From:<\/b> Buck Hodges\nSent:<\/b> Tuesday, November 18, 2014 8:15 PM\nTo:<\/b> Brian Harry\nSubject:<\/b> FW: Mitigations and Rollback steps<\/p>\n

    These were the two things we did. The first made the service usable for most everyone (SPS was down due the calls to storage taking forever for things like roaming settings). The second made it usable by the rest (if your user prefs weren\u2019t cached, you got a 500 when the web page loaded \u2013 I had a couple of accounts affected that way, which we used to verify that change addressed the problem). We\u2019ll be changing the code to be resilient there. Due to past problems, we at least had the switches to be able to turn them off.<\/p>\n

    Some users may still have some issues intermittently with accessing files, work item attachments, git pushes.<\/p>\n

    This pair of changes put us in pretty good shape relatively quickly. A lot of other services didn\u2019t fare as well.<\/p>\n

    Reaction time was not bad with room for improvement. First alert at 4:58 PM PST. First customer report via Twitter 5:03 PM PST (from Australia). Bridge spun up by 5:10 and had a bunch of people on it within a few minutes. Took us a little while to get to the storage issue. Applied the first change at 5:40 and figured out issue to address with second change an hour later. We actually have a report that makes the storage issue pretty clear. We need to get storage represented our devops overview report to narrow it down faster.<\/p>\n

    We\u2019ll leave these mitigations in place until we get clear indications that things are good, both from azure and from pinging our storage accounts (probably over night).<\/p>\n

    Buck<\/p>\n

    ——– End mail thread ——–<\/p>\n

    Again, it is not my intent to overlook the seriousness of any outage.\u00a0 But one of the things that I\u2019ve learned is that building the culture in the team and leaders who are quick, decisive, drive for root cause and are always asking how we can do better in the future is one of the most important things any team running a mission critical service can do.\u00a0 I\u2019m not proud of what happened last night.\u00a0 But I\u2019m very proud of how the team handled it.<\/p>\n

    Hopefully there\u2019s something you can extract from this that will help you run better services too.<\/p>\n

    Brian<\/p>\n","protected":false},"excerpt":{"rendered":"

    We had an incident on VS Online yesterday, starting at 1 AM UTC (5:00 PM PST).\u00a0 The incident was caused by an underlying Azure storage incident affecting many stamps in several regions.\u00a0 I don\u2019t know the root cause of that yet but I\u2019m sure I\u2019ll find out. I want to start by apologizing for the […]<\/p>\n","protected":false},"author":244,"featured_media":14617,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[8],"class_list":["post-273","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","tag-vsonline"],"acf":[],"blog_post_summary":"

    We had an incident on VS Online yesterday, starting at 1 AM UTC (5:00 PM PST).\u00a0 The incident was caused by an underlying Azure storage incident affecting many stamps in several regions.\u00a0 I don\u2019t know the root cause of that yet but I\u2019m sure I\u2019ll find out. I want to start by apologizing for the […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts\/273","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/users\/244"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/comments?post=273"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts\/273\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/media\/14617"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/media?parent=273"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/categories?post=273"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/tags?post=273"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}