{"id":31935,"date":"2017-05-16T01:21:01","date_gmt":"2017-05-16T08:21:01","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/?p=31935"},"modified":"2019-02-14T15:51:41","modified_gmt":"2019-02-14T23:51:41","slug":"accelerated-continuous-testing-with-test-impact-analysis-part-2","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/devops\/accelerated-continuous-testing-with-test-impact-analysis-part-2\/","title":{"rendered":"Accelerated Continuous Testing with Test Impact Analysis – Part 2"},"content":{"rendered":"

The previous post<\/a> introduced how – for a given code commit – TIA will select and run only the relevant tests required to validate that commit. Thus, without sacrificing quality, both the testrun and its enclosing CI definition will complete faster. Here is how that translated to reality for one of our teams:<\/p>\n

\"tiablog-part2image\"<\/a><\/p>\n

The top graph plots the time-to-completion for the CI definition\nThe blue curve is the observation when TIA is OFF. No test selection in effect, and hence \u201call\u201d tests are run after each build \u2013 thus, the almost constant time taken for each run of the CI definition.\nThe orange curve is the observation when TIA is ON. Automatic test selection is in effect. Since only impacted tests are selected to run, the overall CI definition completes faster.<\/p>\n

The bottom graph plots the testrun time.\nThe blue curve is the observation when TIA is OFF. No test selection in effect, and hence \u201call\u201d tests are run \u2013 thus, the almost constant time taken for each testrun as well.\nThe orange curve is the observation when TIA is ON. Automatic test selection is in effect. Since only impacted tests are selected to run, the overall testrun time is predominantly\u00a0lower.<\/p>\n

Of course, YMMV<\/em>.<\/p>\n

Enable TIA in CI\/PR workflows\n<\/strong>Continuous Integration (CI) is a key practice in the industry. Integrations are frequent, and verified with an automated build that runs regression tests to detect integration errors as soon as possible. As a code base grows and matures its regression test suite tends to grows as well – to the extent that running a full regression test might require hours – slowing down the frequency of integrations, and ultimately defeating continuous integration. In order to have a CI definition that completes fast, some teams defer the execution of their longer running tests to a separate stage in the pipeline. This only serves to further defeat continuous integration.<\/p>\n

DevOps is all about a fast development to delivery pipeline and continuous delivery of value. Releases are happening in days and weeks. In such a DevOps world, there is not going to be any continuous delivery if you do not have your testing right.\u00a0There is no way that we\u00a0are going to have development completed and integrated unless we also have a way to ensure that the development is ready to be entered into an integration pipeline.<\/p>\n

TIA can rescue this situation.\u00a0TIA is supported in the CI and PR workflows<\/a>. TIA will automatically select only the subset required to validate the code being committed.<\/p>\n

TIA has:\n(1)<\/strong> a robust test selection<\/a>\u00a0 – this includes existing impacted tests, previously failing tests and newly added tests.\n(2)<\/strong> safe fall back – for commits and scenarios\u00a0that TIA cannot reason about, it will safely fall back to running all tests (you can see that mentioned in the logs). TIA is currently scoped<\/a> to only managed code, and single box topology. So for e.g. if the code commit contains changes to HTML\/CSS files, it cannot reason about them and will fall back to running all tests.\n(3)<\/strong> configurable overrides – you can run “all” tests at a configured periodicity.<\/p>\n

Now you can, and ought to, make more of your automated tests available to run\u00a0early in the integration pipeline.<\/p>\n

Enable TIA. You have everything to gain.<\/strong><\/p>\n

Caveats\n<\/strong>A couple of caveats when using TIA with Visual Studio 2015 to be aware of:\n(1)\u00a0Running tests in parallel\u00a0– in this case,\u00a0tests will run serially.\n(2) Running\u00a0tests with code coverage enabled – in this case, code coverage data\u00a0will\u00a0not get collected.<\/p>\n

Next post\n<\/strong>We will continue this series in the next post, looking at more details, specifically: manual overrides, and repo considerations. If you would like any other aspect covered in more detail, let us know.\nLooking forward to your feedback.<\/p>\n","protected":false},"excerpt":{"rendered":"

The previous post introduced how – for a given code commit – TIA will select and run only the relevant tests required to validate that commit. Thus, without sacrificing quality, both the testrun and its enclosing CI definition will complete faster. Here is how that translated to reality for one of our teams: The top […]<\/p>\n","protected":false},"author":765,"featured_media":45953,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[226,1,252],"tags":[],"class_list":["post-31935","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ci","category-devops","category-testing"],"acf":[],"blog_post_summary":"

The previous post introduced how – for a given code commit – TIA will select and run only the relevant tests required to validate that commit. Thus, without sacrificing quality, both the testrun and its enclosing CI definition will complete faster. Here is how that translated to reality for one of our teams: The top […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/31935","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/users\/765"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/comments?post=31935"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/31935\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media\/45953"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media?parent=31935"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/categories?post=31935"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/tags?post=31935"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}