{"id":28125,"date":"2017-03-02T20:41:51","date_gmt":"2017-03-03T03:41:51","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/?p=28125"},"modified":"2019-02-14T15:55:48","modified_gmt":"2019-02-14T23:55:48","slug":"accelerated-continuous-testing-with-test-impact-analysis-part-1","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/devops\/accelerated-continuous-testing-with-test-impact-analysis-part-1\/","title":{"rendered":"Accelerated Continuous Testing with Test Impact Analysis – Part 1"},"content":{"rendered":"

Continuous Testing in DevOps<\/h3>\n

In older testing strategies, large software changes were tested as a complete product after a so called “release to QA”, running almost all tests just before release. We know the downsides to that. On the other hand, DevOps is all about a fast development to delivery pipeline and continuous delivery of value. Releases are happening in days and weeks – not in years as they used to. In such a DevOps world, there is not going to be any continuous delivery if you do not have your testing right. Just as we have CI Continuous Integration (CI) and Continuous Delivery (CD), DevOps calls for Continuous Testing (CT).<\/p>\n

Test Impact Analysis (TIA)<\/h3>\n

Can such continuous testing also be fast? Can it still be comprehensive? Should you just run all your tests all the time? At smaller scales running through all your test suites and emitting\u00a0out copious test results may be tolerable, but this not scale. At a larger scale, for testing cycles to be fast, a more sophisticated view of what is comprehensive testing becomes essential. “Relevance” becomes critical – only run the most relevant tests and only report the relevant results.<\/p>\n

The Test Impact Analysis (TIA) feature specifically enables this – TIA is all about incremental validation by automatic test selection<\/strong>. For a given code commit entering the pipeline TIA will select and run only the relevant tests required to validate that commit. Thus, that test run is going to complete faster, if there is a failure you will get to know about it faster, and because it is all scoped by relevance, analysis will be faster as well.<\/p>\n

\"tia\"\n<\/a>TIA does come with a little overhead of its own (in order to build and maintain the mapping), and so is best applied only in cases where a test run itself takes a little long to complete (say, > 15 mins).<\/p>\n

Enabling TIA<\/h3>\n

TIA is supported via the Version 2.* of the Visual Studio Test task. If your application is a single tier application, all you need to do is to check \u201cRun only impacted tests\u201d in the task UI. The Test Impact data collector is automatically configured. No additional steps required.<\/p>\n

\"vstestv2task\"<\/a><\/p>\n

If your application interacts with a service in the context of IIS, you need to additionally configure the Test Impact data collector to run in the context of IIS (via a .runsettings file). Here is a sample that does this configuration:<\/p>\n

<?<\/span>xml<\/span> version<\/span>=\"1.0\"<\/span> encoding<\/span>=\"utf-8\"?><\/span>\n<<\/span>RunSettings<\/span>>\n<\/span>  <<\/span>DataCollectionRunSettings<\/span>>\n<\/span>    <<\/span>DataCollectors<\/span>>\n<\/span>      <!--<\/span> This is the TestImpact data collector.<\/span>-->\n<\/span>      <<\/span>DataCollector<\/span> uri<\/span>=\"datacollector:\/\/microsoft\/TestImpact\/1.0\"<\/span> assemblyQualifiedName<\/span>=\"Microsoft.VisualStudio.TraceCollector.TestImpactDataCollector, Microsoft.VisualStudio.TraceCollector, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\"<\/span> friendlyName<\/span>=\"Test Impact\">\n<\/span>        <<\/span>Configuration<\/span>>\n<\/span>          <!--<\/span> enable IIS data collection<\/span>--><\/span>\n          <<\/span>InstrumentIIS<\/span>><\/span>True<\/<\/span>InstrumentIIS<\/span>><\/span>\n          <!--<\/span> file level data collection<\/span> --><\/span>\n          <<\/span>ImpactLevel<\/span>><\/span>file<\/<\/span>ImpactLevel<\/span>>\n<\/span>          <!--<\/span> any job agent related executable or any other service that the test is using needs to be profiled.<\/span> -->\n<\/span>          <<\/span>ServicesToInstrument<\/span>><\/span>\n            <<\/span>Name<\/span>><\/span>TeamFoundationSshService<\/<\/span>Name<\/span>><\/span>\n          <\/<\/span>ServicesToInstrument<\/span>>\n        <\/<\/span>Configuration<\/span>>\n<\/span>      <\/<\/span>DataCollector<\/span>>\n<\/span>    <\/<\/span>DataCollectors<\/span>>\n<\/span>  <\/<\/span>DataCollectionRunSettings<\/span>>\n<\/<\/span>RunSettings<\/span>><\/span><\/span><\/pre>\n
\"\"<\/a><\/p>\n
Reporting<\/h3>\n
TIA is in integrated into existing test reporting at both the summary and details levels, including notification emails.<\/p>\n
\"report\"<\/a><\/p>\n
Robust Test Selection<\/h3>\n
But why should you trust TIA? How do you know that quality has not been compromised in all of this? To answer that we need to look at 2 aspects:\n(1)\u00a0what are the tests that get automatically selected\n(2)\u00a0policies that can condition test selection.<\/p>\n
TIA will look at an incoming commit, and select the set of relevant tests \u2013 these will have 3 components\n(1)<\/strong> The existing tests<\/strong> impacted by the incoming commit.\n(2)<\/strong> Additionally, it will also select previously failing tests<\/strong> \u2013 if not, then over the course of several commits some earlier failing test case might just get lost \u2026 Therefore, TIA will keep track of tests that failed in the previous build and include them in the selection.\n(3)<\/strong> It will also include newly added tests<\/strong> \u2013 what if your commit contains new tests? Such tests could uncover product bugs, right? So, TIA will select newly added tests as well.<\/p>\n
Taken together, this complement makes for a robust selection of tests. Additionally, we allow (and recommend) running\u00a0all tests at a configured periodicity.<\/p>\n
Scope<\/h3>\n
TIA is presently scoped to the following:<\/p>\n
    \n
  • Supported\n
      \n
    • TFS 2017 Update 1 onwards and on VSTS<\/li>\n
    • Version 2.* of the Visual Studio Test task in the build definition<\/li>\n
    • Build vNext, with multiple VSTest Tasks<\/li>\n
    • VS2015 Update 3 onwards on the build agent<\/li>\n
    • Local and hosted build agents<\/li>\n
    • CI and in PR workflows<\/li>\n
    • Git, GitHub, External Git, TFVC repos<\/li>\n
    • IIS interactions (over REST, SOAP APIs), using HTTP\/HTTPS protocols<\/li>\n
    • Automated Tests<\/li>\n
    • Single box topology (tests and application must be running on the same machine).<\/li>\n
    • Managed code (any .NET Framework application, any .NET Service)<\/li>\n<\/ul>\n<\/li>\n
    • Not yet supported\n
        \n
      • Multi box topology (where the test is exercising an app deployed to a different machine)<\/li>\n
      • Data driven tests<\/li>\n
      • Test Adapter-specific parallel test execution<\/li>\n
      • .NET Core<\/li>\n
      • UWP<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
        Next post<\/h3>\n
        We will continue this series in the next post, looking at more details, specifically: manual overrides, repo considerations, using TIA in PR builds, and interaction with Code Coverage. If you would like any other aspect covered in more detail, let us know.<\/p>\n
        In the meantime, start using TIA! Looking forward to your feedback.<\/p>\n","protected":false},"excerpt":{"rendered":"
        Continuous Testing in DevOps In older testing strategies, large software changes were tested as a complete product after a so called “release to QA”, running almost all tests just before release. We know the downsides to that. On the other hand, DevOps is all about a fast development to delivery pipeline and continuous delivery of […]<\/p>\n","protected":false},"author":765,"featured_media":45953,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[226,1,252],"tags":[],"class_list":["post-28125","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ci","category-devops","category-testing"],"acf":[],"blog_post_summary":"
        Continuous Testing in DevOps In older testing strategies, large software changes were tested as a complete product after a so called “release to QA”, running almost all tests just before release. We know the downsides to that. On the other hand, DevOps is all about a fast development to delivery pipeline and continuous delivery of […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/28125","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/users\/765"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/comments?post=28125"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/28125\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media\/45953"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media?parent=28125"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/categories?post=28125"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/tags?post=28125"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}