State of VS 2010 Performance

Brian Harry

Up until Christmas, I was blogging almost every day about the progress we’ve made on addressing the Beta 2 feedback that VS 2010 had significant performance issues.  Since then, our efforts have hardly taken a break.  I saw active investigations all the way up to Christmas eve and then a bit of a lull until Jan 4th when activity resumed in full force.

The improvements are so many and so dramatic that I really couldn’t enumerate them all here.  I think the best way to summarize it is to talk about the feedback we are getting from people as they use new builds.  Since our renewed performance effort started, we have produced 3 Super Limited Community Technology Preview releases (SLCTPs – yeah, a mouth full :)).  These have been delivered to virtually anyone who has expressed performance dissatisfaction, and for whom we could find contact information.  With each one we contact as many recipients as we can and solicit feedback on the changes.

About a week ago, we delivered SLCTP3.  A few hundred people (outside Microsoft) have received it and so far we have close to a hundred survey responses from them.  We also made a huge push to get everyone in Developer Division upgraded to a January build.  I think, as of late this week, we managed to get over 80% of the division upgraded.  Lastly, we’ve reached out to partner teams like Blend, Exchange, CodePlex and Oslo to upgrade and give us feedback on the new builds.  The difference between now and Beta 2 is staggering.

Before I go into the details, let me remind you of the results we got from our initial Beta 2 surveys.

Our initial external customer survey said that approximately 30% of Beta 2 testers where somewhat or very dissatisfied with Beta 2 performance (the earler version of the “Performance” chart below).  After seeing that and running an internal survey of dogfooders, we found that about 70% of internal users were somewhat or very dissatisfied with Beta 2 performance.

Now, at SLCTP3, it looks very different.  Our external LCTP3 results look like this.  And remember, all of these results are from people who complained about Beta 2 performance (not the 70% of people who didn’t).  We didn’t give any of the LCTPs to or survey people who were already happy.

And our internal dogfooder survey looks like this.  The “improved” questions are relative to Beta 2.  The Ready To Ship and Feature Satisfaction questions are absolute.  The performance feature satisfaction question was 70% somewhat or very dissatisfied after Beta 2.

As you can see, the picture looks totally different.  The very dissatisfied numbers are nearly gone and the total dissatisfied are way better.  I have no illusion that we can make everyone happy – we could work on it for 2 more years and still have a few people who would like even more but I feel like we have really turned the corner.  We aren’t done but we are now ready to go out with a broad Release Candidate and solicit feedback from thousands rather than hundreds.

Beyond the statistics, some of the verbatim customer comments tell an even more compelling story…

• The LCTP-3 build appears to resolve all of the issues I have reported pertinent to the normal VS2010 Beta 2 r elease: the uninstall and reinstall procedure worked without any anomalies, including full installation of the LCTP-3 version of MS Help Viewer 1.0; … I have seen no issues with LCTP-3 in the brief work it so far that suggests it is not fully ready for full production.
• The performance is acceptable now and I would consider the product generally shippable…
• Performance on SLCTP3 is ten times better than that of Beta. Previously in Beta I’d crash every 1.5 hours. Since moving to SLCTP3 I have not received any crashes to date.
• I’ve been running the SLCTP3 for about 3.5 hours today and its been amazing. Previously I would have crashed at least 2 times and had insane perf issues.
• It’s such a huge improvement so far from a performance perspective that I’m excited to work with it again.
• I’d say the performance is about equal or maybe slightly better in some scenarios than VS2008. Exciting compared to the not so good performance of beta 2.
• I reported performance problems with navigating the IDE (menus, windows, etc.) over RDP, and this build is a big improvement over beta 2. VS2008 still feels snappier when compared side-by-side on the same VM, but the performance doesn’t bother me in this build of VS2010.
• The Silverlight designer (still working on a Silverlight 3 project) seems to load MUCH faster than with Beta 2.  It’s a night and day difference.

And not to ignore any negative feedback:

• Build time is 10-15% longer for the same solution compared to 2008
• Compilation of WPF project is >500% slower on VS 2010 SLCTP 3
• Intellisense pop-ups are slow to pop up
• Was using ReSharper 5.0 but its way too slow and buggy at the moment. It makes the environment very fragile and unstable.
• When its a fresh install it feels great. However, if you have a lot of add-in’s it seems to start to get a little sluggish on startup and when you perform certain actions.

We are following up on all of these with survey respondents.

We are not done with performance or functional improvements but I believe we have addressed all of the major issues and are now ready to make sure we’ve hit all of the combinatorics, edge cases, etc.  That’s what a Release Candidate will help us do.  In parallel with getting an RC ready, there are a few performance areas where we are still working diligently.  Probably the biggest one is solution build performance.  Many build scenarios are faster than VS 2008 but we’re also still tracking a number of applications where it is slower (generally between 15 and 20%).  We are also putting some finishing touches on some remaining editor/WPF painting and responsiveness issues.  I think even without them you’ll find the RC performance to be quite good but we’ll continue to polish it while we wait for RC feedback.

We’re working on getting the release candidate ready now and I’ll let you know as soon as it is available.

It’s been a hectic couple of months working through all of the feedback and resulting issues.  It’s not quite what I was expecting to be doing but that’s the way it goes.  I’ve got more fodder for posts on things we’ve learned along the way and I’ll try to get a few of them written over the next month or so.

Thanks for helping us work through this and deliver a product that you are going to love,

Brian

Brian Harry Corporate Vice President, Cloud Developer Services

Follow