{"id":9651,"date":"2007-03-09T08:06:52","date_gmt":"2007-03-09T08:06:52","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/bharry\/2007\/03\/09\/dogfood-server-upgrade-end-of-week-1\/"},"modified":"2018-08-14T00:34:15","modified_gmt":"2018-08-14T00:34:15","slug":"dogfood-server-upgrade-end-of-week-1","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/bharry\/dogfood-server-upgrade-end-of-week-1\/","title":{"rendered":"Dogfood Server Upgrade &#8211; End of Week 1"},"content":{"rendered":"<p>The first week of the Orcas Dogfood server upgrade will end today.&nbsp; It&#8217;s been a fantastic (if hectic) week.&nbsp; After the initial spate of issues we hit Monday, it quieted down pretty quickly.&nbsp; We got the significant issues fixed on Tuesday and have been making small performance patches all week.&nbsp; We&#8217;re down to the last half dozen or so issues to investigate and will finish that up at lower priority over the next few weeks.<\/p>\n<h3>Before &amp; After<\/h3>\n<p>We now have enough data to start to do some semi-meaningful before and after comparisons.&nbsp; I picked a set of server requests to compare.&nbsp; I chose them based on a few criteria:<\/p>\n<ul>\n<li>Aggregate cost of the request (over the period of several days) is in the&nbsp;top 10<\/li>\n<li>The average execution time of the request is in the top 10<\/li>\n<li>The numbers looked suspicious to me in some way \ud83d\ude42<\/li>\n<\/ul>\n<p>Given that, here&#8217;s some comparative results.&nbsp; These results show average duration of the request during the month of February compared to the average time since the upgrade (with some tinkering to account for patches that we&#8217;ve made).<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/8\/2019\/02\/image%7B0%7D%5B3%5D.png\"><img decoding=\"async\" style=\"border-right: 0px;border-top: 0px;border-left: 0px;border-bottom: 0px\" height=\"441\" src=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/8\/2019\/02\/image%7B0%7D_thumb%5B1%5D.png\" width=\"401\" border=\"0\"><\/a> <\/p>\n<p>As you can see there are some very healthy improvements.&nbsp; The most concerning regressions are Upload and Download.&nbsp; We will be investigating those shortly.&nbsp; We have been inclined to disbelieve those results as we really didn&#8217;t change that code in Orcas but I think we have enough data now to show that something is afoot.&nbsp; We believe ReadIdentityFromSource is suffering from some ActiveDirectory latency issues but we don&#8217;t know for sure yet.&nbsp; ReadIdentity is showing a huge regression in multiples but pretty small in absolute value.&nbsp; It&#8217;s going to require some poking around to understand.<\/p>\n<h3>SQL CPU Utilization<\/h3>\n<p>We expected to see a substantial reduction in CPU utilization based on the changes we&#8217;ve made but we haven&#8217;t.&nbsp; The standard deviation has gotten much less (with no more large spikes) but the average doesn&#8217;t seem to have gone down much.&nbsp; We need a bit more trend data and need to do some investigation.&nbsp; I expect we&#8217;ll learn more about this over the next couple of weeks.<\/p>\n<h3>SQL I\/O<\/h3>\n<p>We&#8217;re going to be starting our detailed I\/O analysis in the next couple of days (now that most of the biggest perf issues have been investigated and addressed).&nbsp; I&#8217;ll share that with you next week.&nbsp; However, I&#8217;ve done some preliminary looking at the I\/O perf counters on the data tier and the results are interesting.&nbsp; I&#8217;m seeing a dramatic drop in reads on both the data drive and the&nbsp;TempDB drive (2X or more).&nbsp; However, I&#8217;m seeing increases in writes to both.&nbsp; The increases in writes to the data drive are small and the increases for the&nbsp;TempDB drive are modest.&nbsp; I think we&#8217;ll know a lot more after the detailed analysis.<\/p>\n<h3>Conclusion<\/h3>\n<p>Overall, things are going really well and I&#8217;m psyched about it.&nbsp; It&#8217;s been a lot of fun the last few days hammering out all of the issues that are hard to find outside a high-scale production environment.&nbsp; I&#8217;m planning on producing my March dogfood statistics next Tue or Wed, so keep your eyes open for that.<\/p>\n<p>Until next time,<\/p>\n<p>Brian<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The first week of the Orcas Dogfood server upgrade will end today.&nbsp; It&#8217;s been a fantastic (if hectic) week.&nbsp; After the initial spate of issues we hit Monday, it quieted down pretty quickly.&nbsp; We got the significant issues fixed on Tuesday and have been making small performance patches all week.&nbsp; We&#8217;re down to the last [&hellip;]<\/p>\n","protected":false},"author":244,"featured_media":14617,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[5,3],"class_list":["post-9651","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","tag-tfs","tag-tfs-dogfood-statistics"],"acf":[],"blog_post_summary":"<p>The first week of the Orcas Dogfood server upgrade will end today.&nbsp; It&#8217;s been a fantastic (if hectic) week.&nbsp; After the initial spate of issues we hit Monday, it quieted down pretty quickly.&nbsp; We got the significant issues fixed on Tuesday and have been making small performance patches all week.&nbsp; We&#8217;re down to the last [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts\/9651","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/users\/244"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/comments?post=9651"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts\/9651\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/media\/14617"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/media?parent=9651"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/categories?post=9651"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/tags?post=9651"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}