{"id":1541,"date":"2014-02-10T09:00:00","date_gmt":"2014-02-10T09:00:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/bharry\/2014\/02\/10\/visual-studio-online-update-feb-10\/"},"modified":"2022-05-10T02:37:05","modified_gmt":"2022-05-10T09:37:05","slug":"visual-studio-online-update-feb-10","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/bharry\/visual-studio-online-update-feb-10\/","title":{"rendered":"Visual Studio Online Update \u2013 Feb 10"},"content":{"rendered":"<p>Yesterday, we completed our sprint 60 deployment.\u00a0 With it comes two new features: Exporting test plans as HTML documents and a permission to control who can create new work item tags.\u00a0 You can read more about the new features on the service release notes here: <a title=\"http:\/\/www.visualstudio.com\/news\/2014-jan-22-vso\" href=\"http:\/\/www.visualstudio.com\/news\/2014-feb-10-vso\">http:\/\/www.visualstudio.com\/news\/2014-feb-10-vso<\/a>.<\/p>\n<p>As you may know, since ~Oct, we\u2019ve had a run of \u201cbad\u201d deployments that caused unacceptable down time.\u00a0 Due to this we made a number of changes.\u00a0 One was to move our deployments to weekends to reduce the number of people that any unexpected interruptions affect.\u00a0 This was the second deployment to happen over a weekend and thankfully went very smoothly.\u00a0 Sprint 59\u2019s weekend deployment did not go well though and we had a good number of people working all day Saturday and Sunday to get the service back to health.\u00a0 While the weekend deployment had the desired effect \u2013 it reduced the impact on customers, it confirmed for us the heavy toll it takes on the team.<\/p>\n<p>In fact, we moved sprint 60\u2019s deployment to this weekend (it was supposed to be last weekend) because we didn\u2019t want to risk having to ask the team to miss the Super Bowl (which, I might add, the Seattle Seahawks dominated).\u00a0 Thankfully this one went smoothly and the number of people who had to work the weekend was comparatively small.\u00a0 However, it\u2019s clear weekend deployments aren\u2019t really going to be sustainable \u2013 in the sense of keeping a talented group of people willing to work 7 days a week.\u00a0 Next sprint we are going to try doing the deployment on a weekday evening.\u00a0 I \u2018m well aware that no time is a good time for an outage.\u00a0 In fact, I even got some customer emails when 59\u2019s weekend deployment experienced issues, but there are less bad times than others.\u00a0 For context, here\u2019s a graph of activity on our service, starting from Sun morning.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/bharry\/wp-content\/uploads\/sites\/8\/2014\/02\/4503.image_thumb_1A47F97E.png\"><img decoding=\"async\" class=\"alignnone size-full wp-image-15584\" src=\"https:\/\/devblogs.microsoft.com\/bharry\/wp-content\/uploads\/sites\/8\/2014\/02\/4503.image_thumb_1A47F97E.png\" alt=\"Image 4503 image thumb 1A47F97E\" width=\"644\" height=\"178\" srcset=\"https:\/\/devblogs.microsoft.com\/bharry\/wp-content\/uploads\/sites\/8\/2014\/02\/4503.image_thumb_1A47F97E.png 644w, https:\/\/devblogs.microsoft.com\/bharry\/wp-content\/uploads\/sites\/8\/2014\/02\/4503.image_thumb_1A47F97E-300x83.png 300w\" sizes=\"(max-width: 644px) 100vw, 644px\" \/><\/a><\/p>\n<p>In the end, the most important this is getting to the point where every deployment is a \u201cnon-event\u201d (meaning no one even notices it happens because it is so seamless) \u2013 both for our customers and our employees.\u00a0 Time shifting the deployment isn\u2019t a fix, it\u2019s a mitigation and while it\u2019s a wise precaution to take, the effect on the team is profound enough that we have to be very careful how much we rely on it.<\/p>\n<p>In that light, we have done a lot of technical work in the past few months to \u201cget back on top\u201d of deployments.\u00a0 It became clear that the scale of the service had outgrown the engineering processes and resiliency we had built in and it was time to do more.\u00a0 Among other things we\u2026<\/p>\n<ul>\n<li>Doubled down on root cause analysis and post-deployment retrospectives<\/li>\n<li>Fed all engineering process learning (both from deployment retrospectives and elsewhere) into an engineering system backlog in order to concretely track progress and to help prioritize.\u00a0 From there we farmed work out into feature team backlogs.<\/li>\n<li>Made significant changes to our pre-deployment testing and scale validation to make it \u201cmore like production\u201d.\u00a0 While it will never be fully like production, it is helping catch more issues than before.<\/li>\n<li>Accelerated work to enable multiple instances for VS Online.\u00a0 In fact, you don\u2019t know it, but we deployed the first \u201cadditional instance\u201d a couple of weeks ago.\u00a0 Ultimately, this will allow us to run fractions of our production load on independent clusters that can be upgraded separately.\u00a0 This way we will not have to subject the entire customer base to the upgrade at the same time.\u00a0 The instance we set up a couple of weeks ago, we have affectionately called \u201cring 0\u201d or (Scale Unit 0 \u2013 SU0) and it will be where all production deployments happen first.\u00a0 SU0 is in the Azure San Antonio data center and SU1 (the instance we\u2019ve been using for a few years) is in the Azure Chicago data center.\u00a0 Ring 0 will be an instance where we only put accounts who \u201cvolunteer\u201d for the risk of additional volatility \u2013 mine will be there <img decoding=\"async\" class=\"wlEmoticon wlEmoticon-smile\" src=\"https:\/\/devblogs.microsoft.com\/bharry\/wp-content\/uploads\/sites\/8\/2014\/02\/8228.wlEmoticon-smile_58CD4724.png\" alt=\"Smile\" \/>.\u00a0 Once deployments are validated on ring 0, they will incrementally roll out across other instances.\u00a0 Eventually we will add the ability for you to choose which instance your account is on and this will give you the ability to locate your account in your region.<\/li>\n<li>Worked to further decouple services to enable more independent deployment of individual services (TFS, Build, Cloud load, analytics, etc).\u00a0 There\u2019s more work to do here but we are making progress.<\/li>\n<li>Implemented a new \u201clock manager\u201d to better enforce some global service policies\/best practices that we saw often being validated based on our root cause analysis.\u00a0 I hope to write a whole post on this one at some point and maybe even share the code.<\/li>\n<li>And much more\u2026<\/li>\n<\/ul>\n<p>I\u2019d like to think that the smooth deployment this weekend was a reflection of the work we\u2019ve been doing but I\u2019d like to see 3 or 4 smooth ones in a row before I\u2019m willing to give too much credit.\u00a0 We\u2019ll continue to work hard on it.\u00a0 I don\u2019t expect to be at a new \u201cplateau\u201d of service stability and excellent for another few months but we\u2019re definitely on the ascent.<\/p>\n<p>Thanks for your patience and understanding as we continue to work to produce the best developer services on the planet.\u00a0 As always, if you see things we can do better, <a href=\"http:\/\/www.visualstudio.com\/support\/support-overview-vs\">please let us know<\/a>.<\/p>\n<p>Brian<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Yesterday, we completed our sprint 60 deployment.\u00a0 With it comes two new features: Exporting test plans as HTML documents and a permission to control who can create new work item tags.\u00a0 You can read more about the new features on the service release notes here: http:\/\/www.visualstudio.com\/news\/2014-feb-10-vso. As you may know, since ~Oct, we\u2019ve had a [&hellip;]<\/p>\n","protected":false},"author":244,"featured_media":14617,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[8],"class_list":["post-1541","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","tag-vsonline"],"acf":[],"blog_post_summary":"<p>Yesterday, we completed our sprint 60 deployment.\u00a0 With it comes two new features: Exporting test plans as HTML documents and a permission to control who can create new work item tags.\u00a0 You can read more about the new features on the service release notes here: http:\/\/www.visualstudio.com\/news\/2014-feb-10-vso. As you may know, since ~Oct, we\u2019ve had a [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts\/1541","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/users\/244"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/comments?post=1541"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts\/1541\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/media\/14617"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/media?parent=1541"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/categories?post=1541"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/tags?post=1541"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}