{"id":23925,"date":"2016-10-10T05:52:40","date_gmt":"2016-10-10T09:52:40","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/?p=23925"},"modified":"2019-02-14T15:56:14","modified_gmt":"2019-02-14T23:56:14","slug":"tracking-branch-health-and-identifying-flaky-tests-in-rm-driven-test-automation","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/devops\/tracking-branch-health-and-identifying-flaky-tests-in-rm-driven-test-automation\/","title":{"rendered":"Tracking branch health and identifying flaky tests in RM driven test automation"},"content":{"rendered":"

In my previous blog post, I had written about how we have a single Release Definition for our team which runs all the test environments in parallel<\/a>.  Now that Release Management supports branch based filters while listing releases, it is very easy to track the health of a particular branch.  Further, with the work the test team has done around making test case history branch and environment aware, it has become significantly easier to pinpoint the checkin which caused a particular test to start failing, and to identify flaky tests.  <\/p>\n

Branch filters<\/h2>\n

The \u201cReleases\u201d view of our team\u2019s Release Definition (named RM.CDP \u2013 which runs a bunch of test automation per checkin) tells a pretty sordid story regarding the test pass rate (see screenshot below).  There are lots of reds, and I have no idea whether we can deploy to prod or not.<\/p>\n

However, note the \u201cBranch: All\u201d control in the upper right corner (highlighted) which was added recently.  This view lists releases across all branches, and I have highlighted a few of the branches that are being worked on).<\/p>\n

\"image\"<\/a><\/p>\n

I want to check the health of the release branch for the previous sprint i.e. releases\/M106 to see whether we can deploy it to prod or not.  I can do this by selecting this value from the dropdown:<\/p>\n

\"image\"<\/a><\/p>\n

This branch looks much healthier, with a lot more greens.  I dig deeper into the latest build from this branch:<\/p>\n

\"image\"<\/a><\/p>\n

I find that the compat tests are broken (i.e. when the version of RM is greater than that of TFS, or less than that of TFS, then we have issues).  Once we rationalize\/fix up these compat tests in releases\/M106, we can deploy releases\/M106 to prod.<\/p>\n

Thus the branch filter feature can be used to easily find the health of a particular branch.<\/p>\n

Analyzing test failures and identifying flaky tests<\/h2>\n

There are usually 3 kinds of test failures that we see in our automation:<\/p>\n

(1) A dev checkin caused the test to start failing<\/p>\n

(2) There is an infrastructure issue which is causing the test to fail e.g. the machine is out of disk space<\/p>\n

(3) The test is flaky<\/p>\n

Here, I will walk you through how to identify each of these scenarios using the cool \u201cView History\u201d feature that the Test team added, which is now Branch aware as well as Environment aware.<\/p>\n

How to find which checkin caused a test to start failing?<\/h2>\n

Lets look at the health of the features\/rmmaster branch (which is our team\u2019s working branch), and dig into the latest release by double-clicking on it.<\/p>\n

\"image\"<\/a><\/p>\n

I focus on the TfsOnPrem environment to start with (which runs the tests on an on-prem TFS installation), and notice that 96.55% of the tests passed.<\/p>\n

\"image\"<\/a><\/p>\n

I click on this number (96.55%) to see which tests failed.  I note that there is one failing test.  Check out the \u201cView History\u201d link on the right.<\/p>\n

\"image\"<\/a><\/p>\n

I now click on the \u201cView History\u201d button to see when it began failing.  In this view, I first filter by Environment, then by my branch of interest (features\/rmmaster).  Finally, I click on the first red bar in the TfsOnPrem environment to see when it began failing.<\/p>\n

\"image\"<\/a><\/p>\n

This gives me the build that caused the problem.  <\/p>\n

\"image\"<\/a><\/p>\n

If I click on the build, I can see the commit that caused the failure.<\/p>\n

\"image\"<\/a><\/p>\n

Is it an infrastructure issue?<\/h2>\n

If the commit seems unrelated, then you need to check if there is an infra issue.  If the test began failing across multiple branches at the same time, then it is likely an infra issue (assuming that the same checkin isn\u2019t made into multiple branches simultaneously).<\/p>\n

In the above example, it turned out that the test actually began failing across other branches (features\/rmmaster and releases\/M106) at around the same time:<\/p>\n

\"image\"<\/a><\/p>\n

It was indeed an infrastructure issue (where the PAT token for the External TFS server had expired, and therefore all tests that were trying to access this TFS server were failing).  This test in fact caused the tests in the releases\/M106 branch to also go red after we deployed M106 to prod.<\/p>\n

Is it a flaky test?<\/h2>\n

If the test has reds and green interspersed for the same branch, and same environment, then it is likely a flaky test:<\/p>\n

\"image\"<\/a><\/p>\n

Conclusion<\/h2>\n

Hopefully this blog has helped explain how we use Release Management\u2019s branch-awareness features to keep our working branch healthy (or at least as healthy as we can \"Smile\").<\/p>\n","protected":false},"excerpt":{"rendered":"

In my previous blog post, I had written about how we have a single Release Definition for our team which runs all the test environments in parallel.  Now that Release Management supports branch based filters while listing releases, it is very easy to track the health of a particular branch.  Further, with the work the […]<\/p>\n","protected":false},"author":123,"featured_media":45953,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[253,226,1,252],"tags":[],"class_list":["post-23925","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure-devops-server","category-ci","category-devops","category-testing"],"acf":[],"blog_post_summary":"

In my previous blog post, I had written about how we have a single Release Definition for our team which runs all the test environments in parallel.  Now that Release Management supports branch based filters while listing releases, it is very easy to track the health of a particular branch.  Further, with the work the […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/23925","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/users\/123"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/comments?post=23925"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/23925\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media\/45953"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media?parent=23925"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/categories?post=23925"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/tags?post=23925"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}