March DevDiv Dogfood Statistics
Time for the monthly installment again 🙂 I’ve been swamped lately and not able to make progress on my “Managing Quality” series. I’ll get back to that shortly.
This is the first monthly dogfood statistics update since the upgrade of the DevDiv dogfood server on 3/3. So, it’s going to be a bit longer than usual.
Monday was a bit hectic and we had one outage for a couple of hours late Wed afternoon. Other than that, the upgrade seems to have gone relatively smoothly. We’re still assessing the overall performance impact on the server. Yesterday and today, the IT team is taking I/O traces so that we can do a before and after analysis of I/O load and patterns and see if there are any remaining issues to look at. It’s going to be another week or two before we can say anything too definitive about changes in CPU utilization – load is random enough that it’s going to take that long to establish a clear pattern in the data.
But… there is quite a lot we can say. Overall performance seems to be much better. At the bottom you’ll find an update on before & after comparisons (with a more complete request list) of aggregate performance by request type. We’re still investigating the degradations and a few that didn’t improve as much as we thought they would. Generally we’re resolving around 3 of them per day. I expect that we’ll be done in the next week or so.
As you look at the size data below, you’ll notice that the number of files and folders is down. This is because we did a bunch of data clean up and scrubbing in the upgrade process. One of the things that jumps out at me in this data is the number of Get operations in the last week (705,682). The highest I’ve ever seen before in a monthly report was about 250,000. If we hadn’t done the upgrade, that may have created undo load on the server. In the data at the bottom you’ll notice that Get is almost 10 times faster on average.
- Recent users: 1,127 (up 42)
- Users with assigned work items: 2,566 (down 122)
- Version control users: 2,145 (up 130)
- Work Items: 191,955 (up 17,617)
- Areas & Iterations: 7,259 (up 155)
- Work item versions: 1,574,453 (up 173,557)
- Attached files: 66,472 (up 6,923)
- Queries: 15,651 (up 1,181)
- Files/Folders: 72,558,448/14,599,432 (down 4,461,032/1,414,759)
- Total compressed file size: 654 GB (up 140 GB)
- Checkins: 184,431 (up 18,683)
- Shelvesets: 8,328 (up 733)
- Merge history: 168,748,294 (up 15,376,432)
- Pending changes: 1,367,233 (up 393,512)
- Workspaces: 4,335 (up 251)
- Local copies: 436,857,001 (up 66,472,972)
Builds & Tests
- Builds: 2,374 (up 392)
- Test runs: 1,655 (up 53)
- Test results: 168,709 (up 2,801)
Commands (last 7 days)
- Work Item queries: 223,579 (up 51,233)
- Work Item updates: 32,260 (up 2,424)
- Work Item opens: 72,209 (down 3,558)
- Gets: 705,682 (up 610,570)
- Downloads: 24,170,464 (up 4,109,528)
- Checkins: 3,702 (down 383)
- Uploads: 358,536 (up 162,532)
- Shelves: 1,159 (up 56)
Before & After – Average Request Duration
In this chart, I show average request execution time before and after the upgrade. The improvement column is (New – Old)/min(New,Old). Red indicates a regression of 20% or more and green indicates an improvement of 20% or more. There are 26 reds and 66 greens. Some of the reds will stay red – either because we intentionally made the trade off to make one thing slower in order to make others faster (as is the case with UpdateWorkspace) or because the method is rarely used and the regression insignificant enough (in the absolute) that it isn’t worth the time. Perhaps it’s bad marketing to put the regressions first 🙂 But then, I’m a “glass half empty” kind of guy when it comes to performance and quality.
Over the next week or two I’ll follow up with updates based on additional bug fixes, perf counter trends and I/O analysis. Let me know if there’s anything else you’d like to hear about. All this goodness is coming your way pretty soon in the Orcas Beta 1 release.