{"id":1631,"date":"2013-12-13T06:13:37","date_gmt":"2013-12-13T06:13:37","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/bharry\/2013\/12\/13\/what-are-off-times\/"},"modified":"2024-03-27T14:07:03","modified_gmt":"2024-03-27T21:07:03","slug":"what-are-off-times","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/bharry\/what-are-off-times\/","title":{"rendered":"What are \u201cOff-times\u201d?"},"content":{"rendered":"

I got called out in email, I think reasonably appropriately, for using the term \u201coff-times\u201d when referring to choosing a better time to do our sprintly upgrades.\u00a0 In a 24×7 continuous service world, there really is no such thing as a \u201cgood\u201d time to have a problem.\u00a0 I\u2019ve written in the past about the extensive lengths to which we\u2019ve gone to ensure that our full upgrade process is \u201conline\u201d and we pretty much never, ever have to take the whole service down for anything.\u00a0 However, there are bad times and worse times to have a problem.<\/p>\n

We have designed our current upgrade schedule to make sure we have developers available during reasonably normal working hours to ensure that, if something goes wrong, the right people are on hand to address the issues and we don\u2019t burn people out with too many long hours and late nights.\u00a0 The result has been a schedule where our upgrades happen during peak hours of usage.\u00a0 For a long time, that wasn\u2019t really a problem because, for the most part, upgrades were very seamless and reliable.\u00a0 In the past few months it has been more of a problem, hence we are looking at adjusting the timing.<\/p>\n

Back to the original observation about \u201coff-times\u201d\u2026<\/p>\n

I love data.\u00a0 I try to share the kinds of data we collect and how we use it.\u00a0 We have a lot of it.\u00a0 In fact, I just saw a mail saying we needed 20 more TB of space to store all of the data we collect about the operation of the service.\u00a0 Although there\u2019s no true \u201coff-time\u201d there are more and less busy times.\u00a0 Here\u2019s a graph our our weekly activity, starting at 12:00am UTC on Monday and going through the week.<\/p>\n

\"Image<\/a><\/p>\n

Our peaks happen at about 11:00am EST (4:00pm UDT).\u00a0 Usage bottoms out around 12:00am EST (5:00am UTC).\u00a0 Weekend days (Sat & Sun) have a similar pattern but are, overall, significantly subdued compared to weekdays.\u00a0 In fact, you can see that the 3rd \u201cpeak\u201d on Friday (and Friday in general) is much smaller than other weekdays; I suspect a sign people are ready for the weekend \"Image<\/a>.<\/p>\n

For the past many months, I\u2019ve been advocating that we work harder to make updates a \u201cnon-event\u201d \u2013 in every sense of the word.\u00a0 Customers shouldn\u2019t notice them.\u00a0 Our developers shouldn\u2019t notice them.\u00a0 Our service delivery team shouldn\u2019t notice them.\u00a0 They just happen and people are pleasantly surprised when new features show up.\u00a0 It\u2019s a statement not only about the reliability and performance but also about the degree of automation and process management on our side.\u00a0 That\u2019s the goal but clearly, in the last few months, we haven\u2019t been meeting that goal.<\/p>\n

We\u2019ll shift our deployment times from morning EST to evening EST for a while and continue to work on making the process better.\u00a0 I\u2019d very much like to avoid having to ask the team to work weekends, even though that would be the least impactful time.\u00a0 I really want to focus on fixing the problem and have this mitigation only be temporary.\u00a0 We\u2019ll certainly keep you all posted as we make progress.<\/p>\n

Brian<\/p>\n","protected":false},"excerpt":{"rendered":"

I got called out in email, I think reasonably appropriately, for using the term \u201coff-times\u201d when referring to choosing a better time to do our sprintly upgrades.\u00a0 In a 24×7 continuous service world, there really is no such thing as a \u201cgood\u201d time to have a problem.\u00a0 I\u2019ve written in the past about the extensive […]<\/p>\n","protected":false},"author":244,"featured_media":14617,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[13,8],"class_list":["post-1631","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","tag-tfservice","tag-vsonline"],"acf":[],"blog_post_summary":"

I got called out in email, I think reasonably appropriately, for using the term \u201coff-times\u201d when referring to choosing a better time to do our sprintly upgrades.\u00a0 In a 24×7 continuous service world, there really is no such thing as a \u201cgood\u201d time to have a problem.\u00a0 I\u2019ve written in the past about the extensive […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts\/1631","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/users\/244"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/comments?post=1631"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts\/1631\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/media\/14617"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/media?parent=1631"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/categories?post=1631"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/tags?post=1631"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}