![\"image\"](\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/8\/2019\/02\/image_2.png\")
<\/a> ![\"image\"](\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/8\/2019\/02\/image_4.png\")
<\/a> <\/p>\nI missed the June Dogfood statistics – sorry about that. This report represents the change since my last dogfood report in May. The big thing that you will observe is that downloads have dropped dramatically (from a peak around ~150,000,000 to ~50,000,000). The reason for this is that we installed a TFS proxy on our corp net and had the majority of users configure their clients to use it. The proxy is 2 machines configured behind an NLB load balancer. The reason we had to make this configuration change was that during peak hours, downloads were reaching over 1,000 downloads per second. The server simply couldn’t service that many requests and the request queue would fill up and start returning "server unavailable" errors to the clients. Adding a proxy allowed us to offload the download volume and keep the request queue from overflowing. We used an NLB proxy "cluster" to avoid having the same request queue overflow problem on the proxy.<\/p>\n
The other "big event" in the past month was a move of our server from our data center in Tukwila to our new data center in Quincy, WA. I wish I could say that went smoothly. Transferring over 8 terrabytes of data several hundred miles and building out new server infrastructure for a mission critical server is a daunting task. We hit quite a few bumps along the way and my back side is still sore from the beatings (admittedly deservedly) I took over it. Fortunately, we shouldn’t have to do such a thing again soon. One of my big learnings from the process though was that we need a better way to simulate our production environment in a non-production test environment. We really needed to test all of the configuration changes we were making on a reasonably accurate simulation before trying it on the live environment. The problem is that, to clone the hardware, it would cost close to $250K – mostly in the cost of the SAN. Anyway, we’ve embarked on a process of creating such a test environment (even if it doesn’t match the hardware exactly). Hopefully this will smooth any further large scale deployment changes we make down the road. Preventing almost 2,500 people from getting their work done is not a recipe for a long and healthy career \ud83d\ude42<\/p>\n
<\/p>\n