Visual Studio 11 Beta Performance Part #2
Welcome back to part 2 of the Visual Studio 11 Beta Performance series. This week’s topic is Solution Load. We received a great deal of feedback from you clearly telling us that it takes too long to get into your code when loading solutions. I would like to introduce Nathan Halstead, from the Visual Studio Pro Team, whose information below describes the work done to improve the Solution Load experience for you.
Improved Solution Load Time in the VS11 Beta
As the user-base of Visual Studio has grown over the years, so has the size and variety of technologies used by Visual Studio. Based on your feedback, we chose to invest in several performance features that would significantly improve both the scalability and usability of working with large solutions in VS11. In this post, I’ll walk you through one of these investments – asynchronous solution loading.
A little technical background…
Before going into too much detail on what we’ve improved, I’ll introduce how VS2010 and previous VS releases loaded solutions. Solution loading is done in three sequential phases, each managed by a central “Solution Load Manager” running on the UI thread:
Step 1: Load Project Factories – for each type of project you have in your solution, we locate the dll containing the Project Factory implementation for that project type. The project factories are responsible for ensuring that each project knows what files, references, folder, configuration settings, etc. belong to a Project object. They also handle validating that any upgrades or compatibility changes have properly been made and that any error conditions are properly handled.
Step 2: Create Projects – once the project factories have been loaded, we iterate through the list of projects in the SLN file, and rely on the project factories to create the projects one at a time. Because this could be a long operation for large solutions, in VS 2010 we added a progress dialog to this phase to provide some feedback about what was happening in the UI:
Step 3: Notify Event Subscribers – After projects are created by their factories, we give any components already loaded in the system a chance to initialize themselves with information about the now-loaded projects. Any component loaded within Visual Studio can subscribe to event notifications from this process, and most of your favorite tool windows, code-editing features, and 3rd party extensions do. The more extensions you have loaded when you open a solution and the larger your solution, the longer this initialization step takes. Because this step can take a variable amount of time, in VS 2010 we added a dialog to indicate that we were processing the final steps for initializing a solution:
In earlier releases of Visual Studio, overall solution load time was largely bound by the time it took to load the required DLLs and project files from disk in the Create Projects phase. As solutions have grown over time, processing time here has increased. However, a much larger portion of solution load time is now spent in the event notification phase, as more capabilities have been added to Visual Studio and 3rd party extensions have become more popular.
Prototyping a solution with VS2010
Although the team understood why solution could take a long time to load, we needed to do a little research to figure out where and how we could have the most impact on product performance. We knew that the capabilities of the product would continue increase over time. Similarly, average solution sizes are likely to increase, so we chose to focus on reducing the amount of computation done as a UI blocking operation during load.
Late in our VS2010 development phase, we asked a few of our developers to implement a prototype to help us better understand the performance characteristics of solution loading. Since the best test-bed we had available to us was our library of test solutions for VS2010, we added some very basic API support to VS 2010 to enable us to interrupt or skip the loading of specific projects by our Solution Load Manager, freeing up the UI thread for experimentation. We wanted to better understand how the product would behave if we chose to defer the loading of projects. Here’s a video of one of the early prototypes in action:
Overall, the performance of the prototype was encouraging. By deferring project loads, we could enable core editing scenarios very quickly. However, we also found that many components in the product, as well as 3rd party extensions had made explicit implementation assumptions that solution loads would always be an atomic operation. As a result, many components created all-or-nothing data structures to represent aspects of the solution they cared most about. Each time a project was added or removed from the solution, these components would re-create their entire global data structures, which could significantly hurt performance. Furthermore, features that helped you navigate between projects, or were meant to operate on a solution level had no means of requesting that additional projects be loaded. This meant that the prototype made features like IntelliSense, Navigate To, and Go To Definition less reliable. Similarly , visual designers or build operations could simply fail to load or crash if dependencies were not properly loaded by the prototype.
While performance of the prototype was pretty good, we also accumulated a long list of product experiences and features that would need further investment if we chose a deferred loading approach to improve in solution load performance. Given the number of features, experiences, and extensions that would be impacted, we chose to schedule the work for the next product version when we could responsibly implement architecture changes to the product.
(As an interesting aside – because we left some of the API hooks in the product for further prototyping and external experimentation, a member of our extension developer community published an extension that uses these APIs to allow you to control your own solution load behavior. If you’d like to experience the prototype behavior for yourself with VS 2010, you can download the extension from the Visual Studio Gallery: http://visualstudiogallery.msdn.microsoft.com/66350dbe-ed01-4120-bea2-5564eff7b0b2)
Engineering a Solution for VS11
Shortly after the launch of VS 2010, feedback on the overall performance of the product clearly indicated that Solution Load times were a top concern – #4 in our first performance survey to early VS2010 adopters, and currently #4 on our UserVoice site for Performance. Starting from the prototype, we began designing a more comprehensive solution to improve solution load time based on what we had previously learned. We liked the deferred loading approach of the prototype, but the number of affected components in the product was quite large. We looked at data from our Customer Experience Improvement Program, and chose to focus on the commands most frequently executed after a solution is loaded. As it turned out, this was almost exclusively limited to code editing and navigation features. We therefore chose a design that would quickly enable editing, code navigation, and working with visual designers for the most recently used projects, while asynchronously scheduling other load tasks. This meant that code editing would work very quickly, and the work needed to support features like solution build, global refactoring, and code analysis would have a little more time to execute in the background to ensure the features would be ready when they were actually used. With the new design, the project load was now executed in two distinct phases – a Modal Loading phase, and a Background Loading phase:
During the Modal Loading phase, we load projects most likely to be needed by the user and block the UI thread to ensure this task has the highest priority. Specifically, we create only the projects that had files left open in the last VS session or projects. We rely on the new Preview Tab feature (discussed here) to help ensure that only projects that you were actively working with in your last session are loaded in this phase of solution load. If any projects loaded during this phase have dependencies on other projects in the solution, we load the dependent projects as well to ensure features like designers and intellisense work. We then schedule background tasks to handle the loading of all other projects in the solution. Finally, we notify components and extensions that the Modal Load phase of solution loading is about to complete, so they can initialize their data models.
In the Background Loading phase, we unblock the UI thread and start processing the background load tasks one at a time. Each time a project is loaded in the background, we notify components and extensions that the project has loaded. If the user requests information about a project that has not finished loading, we immediately load that project and its dependencies (on the UI thread for maximum performance). We made this operation cancellable if you don’t want to wait. Once all project loading tasks have been processed we notify all components and extensions that the Background Loading phase of solution load has completed.
To help maintain a high degree of compatibility with previous releases, project factories or source code control extensions must opt-in to the background loading phase of the solution load. If they do not, the projects they are responsible for will also be loaded in the in our Modal Loading phase.
The net result of this change is that Visual Studio will become responsive VERY quickly, and you can get to work without waiting for the entire solution to finish loading. Using some large benchmark solutions, we profiled the “Time to Responsive” for various types of projects. Here’s some results from a recent test run with some of these solutions:
With Visual Studio now quick to become responsive, we turned our attention to keeping it responsive while projects loaded in the background. We made two key investments here. First, if a user is actively interacting with the IDE (typing, scrolling, clicking, etc.), we temporarily pause processing the background tasks to avoid interfering with the user’s work. When the user eventually pauses to think for more than a few hundred milliseconds, we resume processing of background tasks. This happens frequently enough that there is very little delay in processing of background tasks.
Second, we still had the matter of project load event subscribers being notified each time a project loaded. We had an existing contract that these events would be called on the UI thread. However, in many cases these event calls themselves could be expensive and stall the UI thread for a few hundred milliseconds, resulting in a jittery experience. To solve this problem, we identified all places in our code base that subscribed to project or solution load event notifications, and updated the code to be highly interruptible, and in many cases, scheduled much of the work on a background thread. In cases where we still had to run code on the UI thread, we broke this code up into smaller chunks of work so no perceptible delays would be felt by users trying to work with the product. In some cases, this meant that we had to limit the work done by specific event subscribers. For example, our background intellisense compiler would try to update itself each time a project loaded, and this could be very expensive with projects now loading one at a time. To fix this, we deferred the work until specific actions actually required us to refresh its state. In this case, work was deferred until a new code file was opened, or when the background project load phase had completed.
These changes equally impact a variety of extensions available for Visual Studio and we are working with providers of the most popular extensions to help them identify places in their products where similar updates may be needed.
When all this work was combined with work we did to reduce memory consumption by loading fewer components into memory until they were absolutely needed, the net result was a much faster and more responsive solution load experience, as you can see in this video:
Utilizing your feedback
We have made a lot of updates to ensure you experience a fast and responsive solution load. We have covered many impacted scenarios and have optimized for the most common interactions with Visual Studio. If you experience slower solution load times than you expect, or if some features don’t seem to behave as you expect during load, please let us know! We have published a rich diagnostic mechanism that allows us to pinpoint the issue you are experiencing and send us a report through Connect: the Visual Studio Feedback Tool. The tool will help you quickly capture a trace, screenshots, or other pieces of data our engineers can use to quickly isolate and fix issues you encounter. We’ll combine this feedback with data you allow us to collect by participating in the Customer Experience Improvement Program and submitting PerfWatson delay reports to identify areas for further performance investments.
You can also let us know what you think in the comments section below. We are listening, and appreciate any feedback you have to offer!
 |
Nathan Halstead – Program Manager – Visual Studio Pro Team
Short Bio: Nathan Halstead has been working with the Visual Studio team since 2008 to drive well-performing customer-focused architectural investments inside Visual Studio. Prior to working with the Visual Studio team, he spent a few years developing Business Intelligence software with the Microsoft Office team, a few months developing test tools for Windows Mobile as an intern, and a few semesters developing machine learning software as a Student Research Assistant at Carnegie Mellon University. |
Coming Up Next
In the next post we will cover the changes the team made to improve Debugging. Again, please let me know where you feel we still need improvement, but please let me know where you see noticeable performance improvements as well. I appreciate your continued support of Visual Studio.
Thanks, Larry Sullivan Director of Engineering
Light
Dark
0 comments