{"id":5113,"date":"2008-03-06T19:11:00","date_gmt":"2008-03-06T19:11:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/vcblog\/2008\/03\/06\/approaches-to-testing\/"},"modified":"2019-02-18T18:54:10","modified_gmt":"2019-02-18T18:54:10","slug":"approaches-to-testing","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/cppblog\/approaches-to-testing\/","title":{"rendered":"Approaches to Testing"},"content":{"rendered":"

My name is Martha Wieczorek and I’m a Software Design Engineer in Test on the Visual C++ IDE team. I would like to share with you some different testing approaches that we use on the team and talk about the advantages\/disadvantages of each approach based on my experience.<\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

Testing effectively and efficiently becomes more and more challenging as the size and complexity of our software grows, especially when the software involves a lot of UI interaction.  <\/span>The maintenance costs of test automation have historically been very high.  <\/span>Therefore, the importance of developing a test automation infrastructure that takes advantage of different testing approaches is very important. <\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

First some definitions:<\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

UI level tests:<\/span><\/b>   <\/span>Feature functionality is tested by manipulating UI elements such as windows and controls, i.e.: menu items, buttons or dialogs.<\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

Object model level tests<\/span><\/b>:  <\/span>Feature functionality is tested programmatically by bypassing the UI. <\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

Component level tests<\/span><\/b>:   <\/span>Multiple components are tested together but without the entire system being present. <\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

Unit level tests:<\/span><\/b>  <\/span>Individual APIs are tested in isolation. This approach is mainly used by Developers, and I am not covering this here.<\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

Object Model\/Component level and UI level approaches to test automation are very different.  <\/span>Each has its own focus and goals and one cannot<\/i> <\/i><\/b>replace another.  Replacing UI-approach by Object Model\/Component approach guarantees that you will miss bugs in the target product.  <\/span>Therefore, it is important to have a good balance between Object Model \/Component Level testing and UI Level testing.<\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

Creating Object Model\/Component Level tests that bypass the UI completely helps my team to develop robust and cost-effective tests suites.  <\/span>We have test hooks in the product that exercise the same functionality as that driven by the UI.  <\/span>This makes writing test automation much more efficient, improves feature coverage, speeds test runs (~70% faster than UI level tests), and most important of all it increases the interaction with developers.  <\/span>This collaboration helps testers understand the components\/features better.  <\/span>In turn, this results in better testing methodology.<\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

While Object Model\/Component Level tests are very beneficial, it is important to keep in mind that UI testing doesn’t go away completely.  <\/span>It is necessary for a real simulation on how the customer is going to use the feature.  <\/span>However, building UI tests is far from trivial.  <\/span>UI changes that occur late in the development cycle (ex: button, menu item, or dialogs may change location or appearance), UI synchronization issues (i.e. issues related to machine speed), localized strings, and many other factors increase the probability of false failures.  <\/span>This makes UI test automation a high maintenance activity.  <\/span>As a result, the QA team spends a lot of time investigating non-product related failures.<\/p>\n

<\/span><\/p>\n

<\/p>\n

 <\/p>\n

<\/span><\/p>\n

Designing, implementing, and running automated tests is critically important, but there are many bugs that test automation cannot catch.  <\/span>Taking the time to check features visually, at the end of the product cycle, is very critical.  <\/span>I have seen bugs when UI elements such as dialogs\/controls are cut-off the screen and these situations are not caught through automation.  <\/span>The importance of real world usage of the feature is very important and should be considered in every product cycle. <\/span><\/p>\n

<\/span><\/p>\n

<\/a><\/p>\n

 <\/font><\/p>\n

<\/span><\/p>\n

Thanks,<\/p>\n

<\/span><\/span><\/p>\n

Martha<\/p>\n

<\/span><\/span><\/p><\/p>\n","protected":false},"excerpt":{"rendered":"

My name is Martha Wieczorek and I’m a Software Design Engineer in Test on the Visual C++ IDE team. I would like to share with you some different testing approaches that we use on the team and talk about the advantages\/disadvantages of each approach based on my experience.   Testing effectively and efficiently becomes more […]<\/p>\n","protected":false},"author":289,"featured_media":35994,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-5113","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cplusplus"],"acf":[],"blog_post_summary":"

My name is Martha Wieczorek and I’m a Software Design Engineer in Test on the Visual C++ IDE team. I would like to share with you some different testing approaches that we use on the team and talk about the advantages\/disadvantages of each approach based on my experience.   Testing effectively and efficiently becomes more […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts\/5113","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/users\/289"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/comments?post=5113"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts\/5113\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/media\/35994"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/media?parent=5113"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/categories?post=5113"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/tags?post=5113"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}