Testing the Visual C++ IDE
Alvin Chardon, VC++ IDE QA member here. I write to our readers today, not with a specific topic in mind, but with a few different thoughts collected during the Orcas product cycle. Hopefully, by the end of this narrative we can take that collection of ideas and form a coherent statement. Don’t worry, I’m sure it will all fit together as nicely as native to managed transitions in COM Interop C++CLI code targeting Excel…don’t ask. So back in the real world, I would like to start by giving you a brief introduction to Alvin. I am a Puerto Rican Software and Electrical Engineer (no, the diploma doesn’t say Puerto Rican), that has been in Microsoft for the past 5 years. All this time my career has been inside the Visual C++ Team. About a year and half ago I moved from the Visual C++ Compiler Team to the Visual C++ IDE Team. And let me tell you, it has been quite a change. C++ is a great language, and being a .NET fan myself, it was an incredible experience to be part of the Visual Studio 2005 release which delivered C++CLI. Testing a new language can be quite a challenge, but by definition, compiler testing is so much easier than UI (User Interface) testing. I have great passion for user experiences, human/computer interactions, software usability, functionality and over all, software quality. Thus I took on the challenge of testing the Visual C++ IDE for a change. A functional IDE in which to write code is such an integral part of the development experience, that some even consider it as important as the compiler. I know most hard core programmers out there prefer notepad-type editors, I know I myself sometimes do, but try consuming the Excel API with C++CLI using notepad and the Visual Studio Command Prompt. (“Again, with the Excel?”, you might ask). I guess it is something that has been on my mind this week. Why? You’ll have to wait. It will all make sense later, I promise.
The Visual C++ IDE is great, but we know it could be even better. One of the toughest challenges facing the VC++ IDE QA Team is the testability of the product. During Orcas, we have spent a lot time thinking about this problem. A good software architecture will have a clear boundary between the code that belongs to the UI and the code that belongs to the engine. Having said that, in Orcas we opted for designing an internal API model to expose the engine’s functionality through test hooks. This enables us to reach into the VC IDE at the component level. Why component-based tests? Because they are more robust, easier to write and more maintainable than UI tests. Don’t get me wrong, I think UI tests are still an integral piece of the quality puzzle we call IDE Testing. Optimally, you can use component-based tests to make sure your engine and data are correct and then use UI tests to verify the engine and data are correctly and accurately connected to the user interface. Scenario testing can be done with either, or a combination of both. Providing your QA team with an arsenal of testing frameworks that include both methodologies, is the key to enabling testability in your product. Ideally, they can all coexist on the same testing environment and provide seamless UI and component-based testing. For example, say you want to test that a glyph, that little picture that identifies the type of code construct and its accessibility, is correctly displayed in the Class View. You could use UI to get the symbol name and glyph description, ask the engine for the code construct type of that symbol, and then validate the correct glyph was loaded and displayed. In the Orcas timeframe, we managed to enable this sort of seamless test environment in the Visual C++ IDE, (or we will once I integrate that little code change into our test library).
One component for which we implemented one of the richest API libraries was IntelliSense. Ah…I can feel readers waking up and rocking their seats at the sound of the word: IntelliSense. Yes, IntelliSense, your friend and foe, lover, hater, friend and enemy. You just can’t live without it when it works to your satisfaction and you just wish you could turn it off when it doesn’t. One of the toughest challenges for IntelliSense testing, is not only correctness, but the sort of data flow and time bound issues that make our VC++ IDE hang or crash. Well, in Orcas we recently finished the development of a spanking new IntelliSense stress test harness. You can input any solution, project or what would have been a compiler command line and stress the life out of it. We identified different types of tests, some for stressing reads from our store and some to stress writes into our store. The harness is smart about typing code in the editor in different styles and executing every single possible IntelliSense operation on their allowed C++ constructs. We measure the execution time, we look for parser Access Violations, failed operations, hangs, crashes, processor CPU % peaks, etc. Since the technologies being used are targeted at component-level testing, the harness produces consistent repros (VC slang for steps to reproduce an issue), provides for a debugging environment where it doesn’t matter if the IDE has focus or not, even when typing, and still gives us real metrics because the operations get executed through the same code paths as when a user requests them. Oh, and at the end, it uses Excel to plot all this information and .NET to send a report out to the team. So you can imagine why my week has been about C++, .NET, COM Interop, and Excel and how stress filled it has been…I mean…stress tests filled. Testing improvements like this stress harness are a great big step towards making the quality of our IDE even better. It has been my personal experience as tester and user of the Orcas Visual C++ IDE that IntelliSense has gotten significantly better. We have gained great momentum during the Orcas product cycle which will keep building up after a great Orcas release towards an even better and mind blowing Orcas + 1 release.
I almost forgot to summarize the end result of our narrative. So what do you get when you add Alvin, C++, IDE, engine level testing, IntelliSense and stress?
Un día en la vida del Visual C++ IDE QA Team
Till next time.