A comment on performance
If you check out our User Voice site, 4 of the top 5 voted items have to do with performance. Over the next few months I’ll comment about some of the work we are doing around VS performance (and there has been a lot). But I just got some data that I wanted to share. About half way down the second page (with 73 votes currently) is a suggestion “Improve Test Manager performance”. This is feedback we’ve heard over the past year or so and have done a lot of investigation and work to improve it. As usual, when you start digging in to this kind of stuff, you find some bone headed things that are easy to fix and then lots of little improvements that eventually add up. I just got our latest perf report on our MTM performance work and figured I’d share the details. In total, we are tracking 38 perf tests for MTM and related scenarios and measuring them both in LAN and WAN scenarios. Let’s talk about LAN first. If, for each scenario, I take the time to do it in MTM 2010 and divide that by the time to do it in MTM 11 and then average those results, I conclude that MTM 11 is 4.6 times faster than MTM 2010. 4.6 times is a pretty massive difference and you should notice it. If instead I sum the times of all the scenarios in MTM 2010 and divide that number by the sum of the times of the same scenarios in MTM 11, I conclude that MTM 11 is 2.8 times faster than MTM 2010. Still a pretty massive improvement. This second way of computing the average has the effect of weighting long running scenarios more heavily than short running ones. For example if I take test 1 that went from 10 seconds to 5 seconds (a 2X improvement) and test 2 that went from 1 second to 0.1 seconds (a 10X improvement), method #1 gives a result of 6X ((2 + 10) / 2) and method #2 gives a result of 2.16X ((10 + 1) / (5 + 0.1)). You decide which method you like better either way, the results are pretty good. And the results are even better for WAN scenarios (lower bandwidth, higher latency) – method #1 gives 4.93 times faster and method #2 give 4.8 times faster. And, of course, we aren’t done yet. There are some regressions since the last build, scenarios still not hitting goal and some downright goofy stuff – like 9 web service calls to submit a new bug (particularly painful in WAN scenarios). I expect by the time we ship, the results will be even better and I hope, when you get a chance to try it out, you’ll really notice a difference in how responsive it feels. Again, I know you want to know about VS perf too and I’ve got plenty to say about it and will be doing so in the coming months. Brian P.S. You might wonder, why I’m writing a detailed post on MTM performance. Aren’t I the TFS guy and not the MTM guy? Well, kind of. I’ve recent been asked to take a more active role our Test Professional planning and delivery strategy. As such, you can expect to see me blogging more about about our plans and progress in that space.
P.P.S. Hmm, Guess I’m gonna need a new blog catagory 🙂
Light
Dark
0 comments