Accelerated Continuous Testing with Test Impact Analysis – Part 3

Pratap Lakshman

June 13th, 20170 0

At its core, TIA collects, and subsequently consults, a map of the dynamic dependencies of each test method as it is executing. As the test method is executing it will cover various methods – the source file in which those methods reside are the dynamic dependencies that get tracked. So, the mapping ends up like the following:

    TestMethod1
        a.cs
        b.cs
        d.cs
    TestMethod2
        a.cs
        k.cs
        z.cs

and so on … Later, when a commit comes in containing say a.cs, TIA consults the mapping and runs only the test methods that had a.cs as their dynamic dependency. It of course takes care of newly introduced test methods (that might come in as part of the commit), and carries forward previously failing test methods as well.

TIA knows about .cs and .vb files and treats all other file types as “unknown” file types. If the commit were to contain any unknown file type, TIA will fall back to running all tests; this is TIA’s fallback logic.

Striking a balance From the perspective of a test selection system, the following properties need to be balanced:

it should be efficient at selecting the relevant tests
it should be precise in that tests that are not affected ought not be selected, and
it should be safe in that tests that are not selected ought not to be affected by changes.

Consider a test selection system that might use block level code coverage data gathered from a previous run to predict tests that are likely to be impacted by code changes made in the current run. While this can be precise about identifying tests, it can be unsafe: any code change not related to a method body might get ignored (e.g., annotations using attributes, etc.). Also, changes like adding a method, removing a method, or overriding a method might go undetected. It might also not address any issues such as library updates, etc.

TIA, on the other hand, tracks dynamic dependencies at the file level. Thus, it might select a few more tests. This make it less precise. However, the approach accounts for changes not related to a method body (annotations, etc.). in conjunction with its fallback logic, this makes it more safe. By collecting dependencies at file level granularity TIA has less data to record, keep updated, and manage. This makes it efficient.

Controlling TIA There are multiple ways to condition TIA’s behaviour.

Through the VSTest task’s UI, TIA can be conditioned to run “all” tests at a configured periodicity. Setting this is recommended, and is the means to regulate TIA’s test selection.
Even after TIA has been enabled in the VSTest task, it can be disabled for a particular build by setting the build variable DisableTestImpactAnalysis to true. This override will force TIA to run all tests for that build. In subsequent builds, TIA will go back to optimized test selection.

As already mentioned when TIA opens up a commit and sees an unknown file type, it falls back to running all tests. While this is good from a safety perspective, tuning this behaviour might be useful in some cases. Here are a couple:

Consider a single repo containing projects from multiple teams (perhaps belonging to a single product family). A team might have a build definition that gets triggered when a commit happens anywhere within the repo, and then run tests only belonging to the team. In this case, TIA’s fallback logic can be conditioned to trigger only if unknown files are committed within certain paths in the repo by setting the TIA_IncludePathFilters to only the relevant paths within the repo using the minimatch pattern. Multiple values can be provided using the semicolon as a separator. By default, there are no path filters explicitly configured, and there is an implicit include of all files in the repository. As soon as you specify an explicit path filter the implicit include of the entire repo no longer applies to the trigger.
Consider changes happening to certain file types that you know do not influence the outcome of tests – say commits happening to some markdown files, .csproj files, etc. In this case TIA can be conditioned to ignore changes to certain file types. For instance, if you want the TIA engine to ignore changes to .csproj files, set the build variable TIA_IncludePathFilters to the value "!**\*.csproj". The value is specified using the minimatch pattern, and multiple values can be provided using the semicolon as a separator.

Trusting TIA To use TIA, one needs to trust it. Here are a couple of ways to validate and grow trust on TIA’s robust test selection.

Manually validate the selection – as the developer who knows how the SUT and tests are architected, one could manually validate that TIA indeed selects the right set of tests (the test results page will list the tests that were run, and the tests that were not impacted).
Run TIA selected tests and “all” tests in sequence – in a build definition, have 2 tests tasks: run impacted Tests (T1), and run “all” tests (T2). If T1 passes, notice that T2 passes as well. If there was a failing test in T1, notice that T2 reports the same set of failures.

Enable TIA in your PR workflow You no longer need to sacrifice the comprehensiveness of your test suite just so that your CI definitions complete fast. Go ahead and “Shift Left” and have a comprehensive test suite that can catch integration errors as soon as possible, and have them run as part of the PR build – let TIA’s test selection take care of the performance.

Enable TIA in your PR workflow. Commit with confidence!