Hi, my name is Jianhua Li, and I am a VC++ IDE QA. Today I am going to talk about VC++ IDE / Design Time stress testing.

What is stress testing?

Design Time Stress (DTS) is part of a broader set of reliability quality criteria focused on measuring an application’s robustness, availability and reliability under stressful conditions.  It’s scoped to Design Time Components which are components exercised by the user when working in the IDE, for example, browsing or intellisense.

Why do we need to “stress” our product?

Basically, we want to make sure our product, VC++ IDE, works reasonably under critical conditions, which is far worse than real world coding. By pushing our product to its limits, we can also find many product bugs that can’t be found by “normal” test cases.  For example a stress test can expose many product crash/hang problems due to various racing/timing issues; subtle memory leaks can also be detected since it will be accumulated and enlarged during the extensive long time testing.  Third, stress testing provides a sufficient simulation of real world programmer’s continuous coding experience. Because of the nature of stress testing, a several hour run of stress testing can be adequately equivalent to several days of real world’s testing, or several weeks’ programmer coding. 

How to we perform stress testing?

When we implement our test infrastructure work for stress testing, we considered the following questions:

·         What scenarios should we stress?

Inside the Visual Studio product division, different teams have invested on different stress scenarios, for VC++, our favorite is IntelliSense! We believe Intellisense is one of the key design time features of VC++ for our customers. Just think of how often you will use AutoComplete, MemberList, ParameterHelp, and QuickInfo on your daily coding work. 

·         What are the requirements/goals for stress testing?

Since this is stress testing, the basic requirement will be “continuous running over a long period without any IDE performance degradation and memory leaks”. In practice, this goal will also guarantee that for a real world programmer, after several weeks coding, the intellisense experience will be as smooth as when coding began. There shouldn’t be any delay for the new intellisense operations.

·         What types of stress testing should we test?

VC++ design time stress contains 2 different test types: Stress Store Reads, and Stress Store Writes.

For “Stress Store Reads” test type, as its name implies, we just read data from the intellisense “store” by performing intellisense operations without editing the code.

For “Stress Store Writes” test type, we will force writes to the intellisense store by automated editing of the code, e.g., deleting and re-typing in every line on a file. After the intellisense store is updated, we will “read” the updated intellisense store to make sure it produces the expected results.

Based on the above design requirements, we implemented a fully component level infrastructure for VC ++ IDE stress testing. This leverages our existing component level framework, and uses DTE and other Microsoft internal component level libraries.

Below is the high level general overview of VC++ IDE stress testing.

1.       Stress Test Initialization: Initialize the run settings which includes:

·         stress test solution: the solution on which the stress testing will be performed

·         stress type: Stress Store Reads or Stress Store Writes

·         stress time: how long the stress testing will last

·         stress iteration: how many iterations the stress testing will perform. 

2.       Open the stress test solution file, get the full list of header and cpp files in all projects in the test solution file, and close the solution. 

3.       Start the stress test monitor process, which will monitor the memory usage during stress run.

4.       Open the test solution file again, do the following loop:

·         Open each header/cpp file in the file list obtained from step 2

§  If (stress type == Stress Store Write) do code editing, e.g., deleting and re-typing each line in current opened file. We keep deleting and re-typing every line in current file, without pause during this process, until we are done with the last line in this file. Then we move cursor back to the beginning of current opened file.

§  Get the position of every token in the current file

§  Iterate over every token. For each one, invoke AutoComplete/MemberList/ParameterHelp/QuickInfo as appropriate for that token 

§  Collect the intellisense results on a log file. 

·         If specified stress time is expired or test iteration is reached, quit the loop in this step; otherwise, go to next header/cpp file in the file list, repeat the same steps above.

5.       Analyze the stress log file, get the overall intellisense operation pass rate, average, minimum, and maximum runtime for each intellisense operation. Make sure there is no performance issue found. Also, analyze the stress monitor log file, and make sure there are no memory leaks. All these kinds of results reporting and analyzing are done automatically.

Note:

·         Since our testing is implemented at component level, the file opening, code editing,  and intellisense invocations are very fast. Normally in 1 second, our stress testing can request about 10 or even more intellisense operations, and we will keep this pace/speed during the whole stress test run period, which is about 8 hours. No real world programmers can do this, I bet. ^_^.

·         For stress run result analysis, we don’t verify the correctness for each intellisense operation, the thing we care about in stress testing is the performance over long period run. So the criteria for stress testing are no IDE crash/hang, no abnormal big runtime for any intellisense operation during 8 hours run, and good overall pass rate, e.g., above 80%.

Some interesting data for Orcas SP1 stress testing.

Now let’s see some interesting data for Orcas SP1 stress testing results.

For Stress Store Reads type testing, on Orcas SP1, we have:

§  IDE continues to run for 8 hours without crash/hang.

§  Over 220k intellisense operations are requested during an 8 hour run, and successful intellisense results are returned for over 98% of those intellisense requests. What does this sound like? Let us make a simulation to real world programming.

1.       Assume that during daily coding, an average C++ programmer will request 1 intellisense operation every 5 seconds.

2.       that’s 12 per minute

3.       720 per hour

4.       at 4 hours of actual coding per day  we get 2880 per day

5.       and 14,400 per week (5 day)

§  Based on above assumption, our stress testing is simulating a real world average programmer’s coding of 76 days / 15 weeks! Now let us look at our performance, for 76 days continues coding, the average runtime per intellisense operation is only about 150 ms, and the peak memory for devenv.exe is only 150 MB.

Please give us your feedback on IDE stress testing.

In Visual Studio 2010, we invested in stress testing even more and extended it to include browsing scenarios, for example GoToDefinition and FindAllReferences. Please let us know of any scenarios that you think should be included in our stress testing. Any comment or feedback is welcome!