Microsoft Visual Studio has had a long and successful run over the last decade plus, and much of that success has been through the partnership with our customers along the way. We believe that you can build better products through better conversations, and our feedback channels are intended to help us listen and respond to the things you tell us as you work with our products on a daily basis.

With upwards of 50 million lines of code, broad changes to Visual Studio can take some time to fully incorporate. This can make it seem like we are not listening, but in truth, we spend extensive time reading, responding, and consolidating customer feedback. And we are constantly looking for new ways to provide more frequent updates to the product.

Generally speaking, the earlier we can get feedback, the more valuable it is, and the more likely we are to be able to implement any subsequent changes. So while we always solicit and appreciate feedback, it is particularly important during the early release timeframe. We aggressively pursue feedback from our customers on things such as user experience, performance and feature enhancements. Typically this feedback arrives via channels such as UserVoice, Microsoft Connect, forums, the Visual Studio blog, and surveys.

For specific issues you encounter, we ask you to submit ideas through UserVoice, and bugs using Microsoft Connect.  Today, we would like to share with you some of the investments we have made to improve your bug filing experience as well as bug quality. But before we dive into these investments, let’s look at what happens when you submit a bug on the Microsoft Connect web site.

A bug’s life – what happens when you file a bug?

When you submit a bug today on the Microsoft Connect web site, the bug goes on a journey around the world (literally). Once filed, an initial customer support team in Shanghai attempts to reproduce the issue based on your description and any attachments you upload. Once they reproduce the issue, they look to see whether anyone else has already filed a bug for that issue. If so, they will close your bug as “Duplicate” and include a link to the original bug. If the bug is for some other team outside of Visual Studio, like Windows or Office, the team replies back, giving the proper channel to contact the other team, and closes the bug as “External”.

Once a bug is identified as Visual Studio product issue, the customer support team routes the bug to the appropriate product team, where our engineers begin a deeper investigation. The team prioritizes the issue based on how critical the issue is, how many users are likely to be impacted, and the risk of introducing bugs elsewhere in the product by implementing the requested change. If the issue or change request is considered a high enough priority for the next release of the product, it is then assigned to a developer for a bug fix. As you might expect, bugs and change requests submitted earlier in a product cycle (Beta) are more likely to be accepted by the product team than change requests that are submitted later in a cycle (Release Candidate or later). Things will likely evolve as our update model matures over time, but at present, this is how it works.

Improving repro rates

We mentioned earlier that the first thing our triage engineers actually do is try to reproduce your issue in house. This process can be quite challenging, when you consider the possibilities of customer machine configurations and product interactions. In general, we are able to repro around 75% of the incoming bugs.  For performance or a crash related issues, the rates go down well under 50%. It’s extremely hard to reproduce a hang, a crash and a performance issue, as it often is driven by factors below the surface. Many times, our engineers cannot reproduce the issue with the given description and information, so they reach back out to you for more details about the issue. The information could be very specific, such as asking you about your machine configuration, or much more involved, such as installing a Performance Diagnostic tool to collect data while you reproduce the issue on your machine. In the more complex situations, we end up giving you directions on how to capture such a file, then upload your file to a temporary workspace (one example of this is Connect bug: 555117). Assuming we can get the data we need, the repro rates for performance and crash still only get to around 50%. The process is cumbersome for both you, and for our engineers.

To address the complexity of reproducing issues, we have released the Visual Studio 2012 Feedback Tool, which makes it much easier for you to quickly file reproducible bugs. The tool helps you quickly capture a trace, screen shots, system information, Visual Studio version, and other relevant pieces of data our engineers need to isolate and address the issue. We released an early version of the tool for Visual Studio 2012 Beta, and the results were very encouraging. To date, bugs filed with the Visual Studio 2012 Feedback tool have around 95% repro rates, including those hard-to-reproduce performance and crash bugs. It is also a much easier experience to file a bug, as you work from within Visual Studio, and automatically capture information rather than manually inputting it.

At the end of this post, you’ll find a link to step-by-step instructions on how to use the tool. But first, let’s look at two features which enable you to log more actionable bugs; automatic data collection and the repro recorder.

Automatic data collection

When you submit a bug, the tool automatically collects information about your machine configuration, operating system, Visual Studio versions, and environment. Let’s take a quick look at some of the data we collect in different file types, and how could we benefit from such information.

There are lots of benefits from Visual Studio’s extensibility, but there are occasions where an extension, package or control might destabilize a Visual Studio installation. For example, when we shipped Visual Studio 2010 productivity power tools, a pack of extensions, we discovered that the solution navigator extension could cause Visual Studio to crash under certain conditions. To address similar cases, the Visual Studio 2012 feedback tool automatically collects a log file, called SQMExtensionsLogs. If our engineers suspect that your issue may be related to a known extension, package or control issue, they can look at this log file and decide on the next steps more readily.

The feedback tool captures information about active Visual Studio instances, including information about version and SKU, as well as a rough statistics report about the size and shape of your solution. This report consists of structural information about your solution, such as how many projects in your solution, the kinds of project used, and number of items in each project, but does not include any specifics about your code. The tool also collects operating system details, such as OS version, locale, whether Visual Studio is running under remote desktop, or even whether Aero is enabled or not. Such information is stored in a VSInfo.xml file. This information is extremely helpful for an engineer trying to repro the issue. It is much more useful than just working with a manually typed description of an issue. It is also much more efficient than reaching back out to the customer to request an upload a sample code or solution file (here is an example of the manual upload experience from MS Connect: 650587).

When testing the Visual Studio code base, we try to use various hardware configurations to mimic our customer’s experiences out in the wild. But given all of the possible combinations, it is simply not possible for us to test our code sufficiently for all configurations. In Visual Studio 2010, for example, we have identified some performance issues if you are on specific hardware and we published a list of hardware configurations that commonly encounter problems, and how to trouble shoot them.  To understand this correlation, we had to collect a lot of information from our customers through emails, or back-and-forth communication on MS Connect, and forums. The Visual Studio 2012 feedback tool collects this machine configuration information automatically, and places it in the DxDiagOutput.txt file. Our engineers will use this information to find correlations with less effort, and fix them.

Repro recorder

While we can sometimes reproduce repeatable issues from a set of written repro steps, we can almost always reproduce an issue when a good recording and diagnostic information is included with the description of the issue. Depending upon the problem type you’re experiencing, some recordings are more valuable than others. For example, what we need to understand a performance issue you are experiencing is quite different than what we need when you experience a crash in Visual Studio. To make this process simpler, we have implemented a feature called repro recorder.

If you encounter slow performance with the product, a trace with Windows Events is very helpful for our engineers to decipher the issue. Such trace files help our engineers identify whether the issue is CPU-based, disk or network intensive, or whether the issue is altogether a different program (for more info, you can read about how to do a performance analysis). In Visual Studio 2010, we asked our customers to collect such trace files by installing Performance Diagnostic tool, reading a “How To” file, and then uploading it to a workspace. Now we have a lighter version of this tool, called the Microsoft Visual Studio Trace recorder, which is integrated with the Visual Studio 2012 Feedback tool. Because the tool optimizes trace collection, you can run it for as long as it takes to show the problem, up to about 2 hours. Hence, you can easily capture traces even if the performance issue is not happening consistently. Depending on the type of issues you’re encountering, our engineers use different tools to analyze your trace file in house. For example, if your issue is related with editing responsiveness (such as Intellisense popping up slowly, or delays in characters appearing as you type), we use our RaceTrack tool that allows us to focus on just the time period during typing specific events (if you like to learn more about how we used this tool to improve editing responsiveness, please see Eric Knox’s blog post).

If you are experiencing a crash in Visual Studio, you will see the Watson dialog. The Windows Error Reporting service collects a memory dump of Visual Studio, and asks you whether to send the report back to Microsoft. This reporting is very helpful for us to identify and fix the most frequent crash issues, but we only know about the issue if you send us the error report. Such reports are collected in a central Watson server, and our engineers constantly watch it to identify the top issues and get most of them fixed. In addition to sending the Watson report, you can report a Connect bug for it using the feedback tool. The Visual Studio 2012 Feedback tool allows you to capture the relevant information easily, working with VS Debugger to capture the dump file, and working with Windows Problem Step Recorder to capture your steps. The Connect bug will give you a place to have a conversation about the issue, whereas Watson captures the issue, but there is no follow-up.

If Visual Studio hangs (for example, the editor freezes up for some time), you usually do not get a Watson dialog. Hangs are similar in nature to crashes, but harder to detect since the IDE generally comes out of the freeze. As with crashes, a memory dump would give us insight as to what is hanging the application. Because the Visual Studio process is still available when a hang occurs, it is usually much easier to collect information about a hang than it is for a crash. And, yes, you can still launch the Visual Studio 2012 Feedback Tool even if you have a hung VS instance. (If you are interested in how you can investigate hangs and crashes, please see this blog post).

For an issue that is not related to performance, hangs, or crashes, you can still leverage the repro recorder feature to capture screen shots. As the old saying goes, a picture is worth a thousand words.

Summary

Visual Studio 2012 Feedback tool is a rich diagnostic mechanism that lets us more accurately pinpoint any issue you may experience, and allows you to quickly send us the issue details via Connect. Your bug will be available to you on Connect just as if you filed the bug manually on Microsoft Connect web site. The tool will help you quickly capture traces, screenshots, or other pieces of data our engineers can use to quickly reproduce issues you encounter. You can also add additional files such as sample code, projects, screen shots, etc. Our engineers can use the additional data to identify offending code paths, correlate issues with system settings or machine configurations, or simply get clearer repro steps.

Please download the Visual Studio 2012 Feedback Tool and try it out, to help us make Visual Studio a better product for you! For more details on how to use the tool, refer to the How to use Visual Studio 2012 Feedback Tool.

Thanks,

Selma Ikiz – Program Manager, Visual Studio

Doug Turnure – Principal Program Manager, Visual Studio