What Does a VC++ QA Do Daily?
Hello everyone, this is Smile Wei, an SDET (QA) in the VC Compiler team. This is not only my first blog on vcblog, but also my first ever blog.
As you might have guessed, I’m a new face around here, joining Microsoft only 3 ½ months ago. Here I’d like to give a brief introduction to the daily life as a VC QA and, to answer the following two questions that my friends kept asking me:
How do I like life at Microsoft, especially on the VC team?
What do I usually do every day?
For the first question, I love working at Microsoft, especially as a QA in the VC Compiler team. Usually people might think a QA may not be as technical as a dev, but this is not true in our team. As a QA in the VC compiler team, the test input are not simply numerical data or operations, in most cases they are C/C++ source code. When you write a test, you need to worry about not only how to write test processes with script languages like Perl, but also how to write test input code in C/C++ to capture both the correct behavior (positive test) and the wrong behavior (negative behavior) of the compiler. Same situation when you write regression tests. Creating a new test usually requires a lot of discussions with devs, because we need to make sure our tests will exactly reflect the new features or changes in the VC compiler parser, even some subtle ones.
Having source code as test inputs also increases the difficulty of bug investigations and reproduction. Because of the complexity of the compiler, some compiler bugs only fail under certain types of machine architectures (x86, amd64, or ia64), some bugs only fail under certain runtime environment (native, /clr, or /clr pure) on a particular architecture, while some bugs only fail with a particular cl.exe parameter (among hundreds of parameter combinations) under a particular runtime environment on a particular machine architecture. Usually you’ll feel that half the work is done once a bug is successfully identified and reproduced. That’s why a large portion of my time is spent on bug investigations.
Although this position is so challenging, I love it so much because I can learn so much about the VC compiler. All the details I learn from this position perfectly complement my previous high-level computer knowledge learned in college. Can you imagine how proud I am to easily find the bug in my friend’s code while it has harassed him for several days? That’s why I love this job. J
For the second question, my major responsibility includes the following tasks:
1. Bug investigations on test runs. Microsoft has daily test runs on every product with all existing test suites not only before the product is shipped, but also after. Any failures in the test runs will be investigated as the 1st priority task and get resolved in the earliest possible version.
2. Test fixes and writing regressions. This is a follow-up task of the first one. Once a VC compiler dev fixes a bug, he’ll publish his private built to me for a more comprehensive test even after he has already performed a sniff testing on his own machine. This double assurance mechanism further reduces the risk to introduce new regression bugs by the bug fix. Some extremely crucial bug fix may involve many tests with substantial complexity. For example, to verify a recent fix, I ran a 4 X 3 X 3 X 2 X 2 X 2 test matrix, and each test run in the matrix involved creating and testing a large 23MB PCH. In Microsoft, a regression MUST be written to address the fix if no previous tests were targeting that source code. With more and more regression tests being added, a complex product like Visual Studio might have huge number of tests associated with it. For example, I own a test suite called the “compiler suite”, which contains about 20,000 test cases, this is only one of the 10 test suites of the VC++ compiler team. Please notice that these are only the tests for the VC++ compiler team, not including the tests for the VC++ library team, IDE team, backend team and release team, not to mention other teams in Visual Studio such as the VB and C# teams. The daily test runs on huge number of tests go some way to guaranteeing the quality of Microsoft products.
3. Verify QFE (Quick Fix Engineering) fixes. Yes, your understanding is correct, the QFEs are those hotfixes occasionally popped out and automatically installed on your machine. The usual QFE process is that the support team collets bug reports from our customers, finds a bug and passes it to the dev team. After the dev team fixes the bug, they alert the QA team so we can test it. The QA team will need to perform a two-round verification. The 1st round is before sending the fix back to the customer who identified the bug, and the 2nd round is after the customer verifies the bug fix, to perform a final verification before it’s officially published into hotfixes. Testing QFE fixes is always our 1st priority tasks to solve customers’ problems in the 1st place.
4. Provide cross-team test support. Occasionally I need to provide test support for other teams – to give a different test prospective for the other team. For example, I recently finished two projects falling into that category and did find some bugs. One project was to test the VS 2008 installation process for the release team, and the other one is to test VS 2008 MSDN samples for the library team. I believe this type of test may help find those bugs that might not have be found by the original test team because of their testers’ familiarity with the correct behaviors and processes of their product. This exactly matches the spirit of a well-known Chinese saying “You won’t have the overview of the mountain if you are already inside it”.
5. Develop internal test tools. As I mentioned in the first two responsibilities, Microsoft performs daily test runs on all of its products, and each test run may execute hundreds of thousands or even millions of tests. How does Microsoft achieve this? The trick is that the whole test execution, test result collection and comparison are fully automated with the support of a large set of test tools. Microsoft employees are encouraged to develop all kinds of internal test tools that can improve the test efficiency and some tools may be adopted and widely used in the whole division or even the whole company.
6. Write tests for new features. Currently the VS team has started working on the next delivery of Visual Studio (the version after VS 2008). A QA will be assigned to work out a whole set of tests targeting each new feature. The test process starts in a very early phase of the product cycle rather than after all the coding is done, which I believe sticks to the agile model instead of the traditional waterfall model. The reason we use an agile model is because the later a bug is found in the product cycle, the more cost will be incurred. Currently most of my responsibilities focus on the first five tasks and I believe in my next vcblog you may see more of my work on new features.
7. Other job responsibilities. This includes everything else on my job commitments or not on my commitments. For example, support campus recruiting, make training slides and videos for new employees, interviewing candidates, and yes, write vcblogs. J
Hope you are not bored by my long introduction. J If you happen to have some interest, comments, or suggestions, why not reply to this blog entry – I look forward to interacting with you? If you happen to have more interest on this challenging position, why not come join us? J
Really appreciate your patience and hope all of you and your families are happy and healthy. Thanks.
VC Compiler Team