{"id":9811,"date":"2007-02-04T09:01:51","date_gmt":"2007-02-04T09:01:51","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/bharry\/2007\/02\/04\/managing-quality-part-2-automated-testing\/"},"modified":"2018-08-14T00:34:19","modified_gmt":"2018-08-14T00:34:19","slug":"managing-quality-part-2-automated-testing","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/bharry\/managing-quality-part-2-automated-testing\/","title":{"rendered":"Managing Quality (part 2) – Automated Testing"},"content":{"rendered":"

As I described in my last article, our first layer of quality assessment is build “scouting”, designed to do a shallow but broad pass across the product to determine if it’s worthwhile to proceed with deeper testing.<\/p>\n

Our next layer is more thorough automated testing runs.  Many would call this “functional” testing but words in the testing space get so overloaded.  It includes both API level tests (which some may call Unit tests), UI automation tests and some broader “system integration tests”.<\/p>\n

We divide our automation tests at this level into two categories – Nightly Automation Runs (NARs) and Full Automation Runs (FARs).<\/p>\n

As their name suggests, NARs are designed to be run every day, on every build that is not SelfToast (see part 1 for a description of this term).  Today we have on the order of 3,000 NAR tests.  In our test planning excercise (when we design the test cases we want to run and choose what will be automated) we prioritize our test cases (1, 2, 3).  The NARs are generally selected from the Pri 1 test cases and chosen so that they can run and the results analyzed within a few hours (important if you are going to do this every day).<\/p>\n

FARs are the sum of all automated tests that we have.  For TFS, this amounts to something on the order of 20,000 tests today.  A FAR run takes about a week to run and analyze the results.  As a result, we don’t start doing them until later in the product cycle and we run them less frequently – the closer to the end we get, the more frequently we run them.  Right now, I think we are running them every 2 or 3 weeks.<\/p>\n

For completeness, beyond FARs, we have what we call a Full Test Pass (FTP).  Which is a period of time where we run multiple NAR and FAR runs on a cross section of our test matrix (the subject of a future Managing Quality post) and run our manual test cases.  Last I checked, we had about 10,000 manual test cases on top of the 20,000 FAR cases.  A Full Test Pass takes somewhere from 2 – 4 weeks.<\/p>\n

So, with that background, on to the reports…  Here’s a recent NAR trend report:<\/p>\n

<\/a> <\/p>\n

As you can see, we breakdown each result by the cause of the failure.  These warrant a little discussion:<\/p>\n