My name is Martha Wieczorek and I’m a Software Design Engineer in Test on the Visual C++ IDE team. I would like to share with you some different testing approaches that we use on the team and talk about the advantages/disadvantages of each approach based on my experience.

Testing effectively and efficiently becomes more and more challenging as the size and complexity of our software grows, especially when the software involves a lot of UI interaction.  The maintenance costs of test automation have historically been very high.  Therefore, the importance of developing a test automation infrastructure that takes advantage of different testing approaches is very important.

First some definitions:

UI level tests:   Feature functionality is tested by manipulating UI elements such as windows and controls, i.e.: menu items, buttons or dialogs.

Object model level tests:  Feature functionality is tested programmatically by bypassing the UI.

Component level tests:   Multiple components are tested together but without the entire system being present.

Unit level tests:  Individual APIs are tested in isolation. This approach is mainly used by Developers, and I am not covering this here.

Object Model/Component level and UI level approaches to test automation are very different.  Each has its own focus and goals and one cannot replace another.  Replacing UI-approach by Object Model/Component approach guarantees that you will miss bugs in the target product.  Therefore, it is important to have a good balance between Object Model /Component Level testing and UI Level testing.

Creating Object Model/Component Level tests that bypass the UI completely helps my team to develop robust and cost-effective tests suites.  We have test hooks in the product that exercise the same functionality as that driven by the UI.  This makes writing test automation much more efficient, improves feature coverage, speeds test runs (~70% faster than UI level tests), and most important of all it increases the interaction with developers.  This collaboration helps testers understand the components/features better.  In turn, this results in better testing methodology.

While Object Model/Component Level tests are very beneficial, it is important to keep in mind that UI testing doesn’t go away completely.  It is necessary for a real simulation on how the customer is going to use the feature.  However, building UI tests is far from trivial.  UI changes that occur late in the development cycle (ex: button, menu item, or dialogs may change location or appearance), UI synchronization issues (i.e. issues related to machine speed), localized strings, and many other factors increase the probability of false failures.  This makes UI test automation a high maintenance activity.  As a result, the QA team spends a lot of time investigating non-product related failures.

Designing, implementing, and running automated tests is critically important, but there are many bugs that test automation cannot catch.  Taking the time to check features visually, at the end of the product cycle, is very critical.  I have seen bugs when UI elements such as dialogs/controls are cut-off the screen and these situations are not caught through automation.  The importance of real world usage of the feature is very important and should be considered in every product cycle.

Thanks,

Martha