At its core,\u00a0TIA collects, and subsequently consults, a map of the dynamic dependencies of each test method as it is executing. As the test method is executing it will cover various methods\u00a0– the source file in which those methods reside are the dynamic dependencies that get tracked. So, the mapping ends up like the following:<\/p>\n
TestMethod1\n a.cs\n b.cs\n d.cs\n TestMethod2\n a.cs\n k.cs\n z.cs<\/pre>\nand so on \u2026\nLater, when a commit comes in containing say a.cs<\/em>, TIA consults the mapping and runs only the test methods that had a.cs as their dynamic dependency. It of course takes care of newly introduced test methods (that might come in as part of the commit), and carries forward previously failing test methods as well.<\/p>\n
TIA knows about .cs<\/em> and .vb<\/em> files and treats all other file types as “unknown” file types. If the commit were to contain any unknown file type, TIA will fall back to running all tests; this is TIA\u2019s fallback logic.<\/p>\n
Striking a balance<\/strong>\nFrom the perspective of a test selection system, the following properties need to be balanced:<\/p>\n
\n
- it should be efficient<\/em> at selecting the relevant tests<\/li>\n
- it should be precise<\/em> in that tests that are not affected ought not be selected, and<\/li>\n
- it should be safe<\/em> in that tests that are not selected ought not to be affected by changes.<\/li>\n<\/ol>\n
Consider a test selection system that might use block level code coverage data gathered from a previous run to predict tests that are likely to be impacted by code changes made in the current run. While this can be precise about identifying tests, it can be unsafe: any code change not related to a method body might get ignored (e.g., annotations using attributes, etc.). Also, changes like adding a method, removing a method, or overriding a method might go undetected. It might also not address any issues such as library updates, etc.<\/p>\n
TIA, on the other hand, tracks dynamic dependencies at the file level.\nThus, it might select a few more tests. This make it less precise. However, the approach accounts for changes not related to a method body (annotations, etc.). in conjunction with its fallback logic, this makes it more safe. By collecting dependencies at file level granularity TIA has less data to record, keep updated, and manage. This makes it efficient.<\/p>\n
Controlling TIA<\/strong>\nThere are multiple ways to condition TIA\u2019s behaviour.<\/p>\n
\n
- Through the VSTest task\u2019s UI, TIA can be conditioned to run “all” tests at a configured periodicity. Setting this is recommended, and is the means to regulate TIA’s test selection.<\/li>\n
- Even after TIA has been enabled in the VSTest task, it can be disabled for a particular build by setting the build variable
DisableTestImpactAnalysis<\/code> totrue<\/code>. This override will force TIA to run all tests for that build. In subsequent builds, TIA will go back to optimized test selection.<\/li>\n<\/ol>\nAs already mentioned when TIA opens up a commit and sees an unknown file type, it falls back to running all tests. While this is good from a safety perspective, tuning this behaviour might be useful in some cases. Here are a couple:<\/p>\n
\n
- Consider a single repo containing projects from multiple teams (perhaps belonging to a single product family). A team might have a build definition that gets triggered when a commit happens anywhere within the repo, and then run tests only belonging to the team. In this case, TIA\u2019s fallback logic can be conditioned to trigger only if unknown files are committed within certain paths in the repo by setting the
TIA_IncludePathFilters<\/code> to only the relevant paths within the repo using the minimatch<\/a> pattern. Multiple values can be provided using the semicolon as a separator. By default, there are no path filters explicitly configured, and there is an implicit include of all files in the repository. As soon as you specify an explicit path filter the implicit include of the entire repo no longer applies to the trigger.<\/li>\n- Consider changes happening to certain file types that you know do not influence the outcome of tests – say commits happening to some markdown files, .csproj files, etc. In this case TIA can be conditioned to ignore changes to certain file types. For instance, if you want the TIA engine to ignore changes to .csproj files, set the build variable
TIA_IncludePathFilters<\/code> to the value\"!**\\*.csproj\"<\/code>. The value is specified using the minimatch<\/a> pattern, and multiple values can be provided using the semicolon as a separator.<\/li>\n<\/ol>\nTrusting TIA<\/strong>\nTo use TIA, one needs to trust it. Here are a couple of ways to validate and grow trust on TIA’s robust test selection<\/a>.<\/p>\n
\n
- Manually validate the selection – as the developer who knows how the SUT and tests are architected, one could manually validate that TIA indeed selects the right set of tests (the test results page will list the tests that were run, and the tests that were not impacted).<\/li>\n
- Run TIA selected tests and “all” tests in sequence – in a build definition, have 2 tests tasks: run impacted Tests (T1), and run \u201call\u201d tests (T2). If T1 passes, notice that T2 passes as well. If there was a failing test in T1, notice that T2 reports the same set of failures.<\/li>\n<\/ol>\n
Enable TIA in your PR workflow<\/strong>\nYou no longer need to sacrifice the comprehensiveness of your test suite just so that your CI definitions complete fast. Go ahead and “Shift Left” and have a comprehensive test suite that can catch integration errors as soon as possible, and have them run as part of the PR build\u00a0– let TIA’s test\u00a0selection take care of the performance<\/a>.<\/p>\n
Enable TIA in your PR workflow. Commit with confidence!<\/p>\n","protected":false},"excerpt":{"rendered":"
At its core,\u00a0TIA collects, and subsequently consults, a map of the dynamic dependencies of each test method as it is executing. As the test method is executing it will cover various methods\u00a0– the source file in which those methods reside are the dynamic dependencies that get tracked. So, the mapping ends up like the following: […]<\/p>\n","protected":false},"author":765,"featured_media":45953,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1,252],"tags":[],"class_list":["post-33506","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","category-testing"],"acf":[],"blog_post_summary":"
At its core,\u00a0TIA collects, and subsequently consults, a map of the dynamic dependencies of each test method as it is executing. As the test method is executing it will cover various methods\u00a0– the source file in which those methods reside are the dynamic dependencies that get tracked. So, the mapping ends up like the following: […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/33506","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/users\/765"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/comments?post=33506"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/33506\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media\/45953"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media?parent=33506"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/categories?post=33506"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/tags?post=33506"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}