What has the VC++ Team Been Doing since VS 2005 Shipped?
Hi! My name is Shaun Miller and I’m the test lead for the Visual C++ Release and IDE quality assurance teams. I’ve been around Microsoft for the last 10 years and Visual C++ has been a big part of my experience here.
We’ve been very busy on the Visual C++ team since we shipped VS 2005. Here are some of the activities that we’ve been focusing on since VS 2005.
We spent a considerable amount of time transitioning to new software lifecycle processes. I’ve got a quick summary. For more information, see Carol Grojeans channel 9 talk on MQ (Milestone Quality).
- Eliminate Test Debt (Test Cases/Test Infrastructure). We worked fast and furious on VS 2005 and left some bugs in some tests and we delayed some test infrastructure work. We also made changes to our processes to avoid building test debt for Orcas.
- 70 percent Automated Code Block Coverage. Not all of the Visual C++ binaries have at least 70% block coverage. We’ve done a lot to improve that. It’s important to note that we are testing code blocks, not source lines. There are a lot of blocks that are generated by the compiler (for instance, when using exception handling).
- Complete Whidbey threat models. We completed threat models for VS 2005, but upon review some were not as thorough as we would have liked.
- Start Orcas with no bug debt. This effort was a transition to get more realistic about how much work we can actually do along with a serious amount of bug fixing. This eventually led to blogging a guide for the kinds of bugs that we can handle during the Orcas product cycle.
- New setup technology rollout. We moved from an old setup technology to a setup based on XML.
- Tests Run anywhere – not lab specific. Visual Studio is a big product and that requires a lot of teams. Some teams, like Visual C++, existed before there was a Visual Studio and their tests ran in their own ways. We spent some time to further a long term objective to move to a model where any teams’ tests can run in any lab in Visual Studio.
- BVT 40% Coverage in 1 hr. We increased the coverage of our build verification testing.
- Feature Directory Definition and Implementation. We got a lot more formal regarding planning features. We now produce feature specifications and implementation plans for every feature in Orcas.
- Build Warning Elimination. In VS 2005, the VC++ compiler eliminated some compiler switches which result in warnings when used and issued new warnings. While we fixed the issues that were critical for VS 2005, the non-critical warnings led to very large build logs. So we did some clean-up to make it easier to find the critical things in our build logs.
- Convert our builds to MSBuild. We now use MSBuild to build Visual C++.
- Orcas Quality Gates. We implemented process to improve the quality of features before they are integrated into Visual Studio. The result should provide you with a better experience with pre-release versions of Orcas than you had with pre-release versions of VS 2005.
- Team Foundation. We’re moving to VS Team Foundation for our source control and bug tracking.
We’ve fixed a lot of bugs:
So far, over 200 VC++ bug fixes are going into VS 2005 sp1 http://blogs.msdn.com/vcblog/archive/2006/6/22.aspx.
So far, over 700 VC++ bug fixes are going into Orcas.
We’ve also been working on VS 2003 sp1. We still have a little work left though it is nearing completion.
Of course, we’ve also been working on Orcas (the next product to release after VS 2005). But that story will unfold here on the Visual C++ Team Blog in the coming months.
Visual C++ Release/IDE