Introduction MSTest V2 v1.3.0 Beta2 now supports in-assembly parallel execution of tests – the top most requested/commented issue on the testfx repo.

The feature can dramatically reduce the total time taken to execute a suite of tests. To get started, install the framework and adapter from NuGet. If you are already using MSTest V2, then upgrade to this version.

Try out the sample on GitHub. The GitHub page has details, but here are the results with that sample (YMMV, of course):

Motivation The key motivation is to complete the execution of a suite of tests, within a single container, faster.

Coarse-grained process-parallelization is already supported by vstest. That works by launching test execution on each available core as a distinct process, and handing it a container worth of tests (assembly, DLL, or relevant artifact containing the tests to execute) to execute. The unit of isolation is a process. The unit of scheduling is a test container. The tests within the container were executed in serial.

in-assembly parallel execution of tests (“IAP”) takes this to the next level by providing finer-grained control over parallel execution – it enables executing tests within a container in parallel.

Requirements “IAP” is intended to meet the following requirements: (1) Easy onboarding – make it possible to enable “IAP” for existing MSTest V2 code. For e.g. there might be 10s of 100s of test projects participating in a test run – insisting that all of them make changes to their source code to enable parallelism is a barrier to onboarding the feature. (2) Fine grained control per assembly – there might still be certain assemblies, or test classes or test methods within the assembly, that might not be parallel-ready. Make it possible for such artifacts to opt-out. Conversely, there might be only a few assemblies that want to opt-in – that should also be made possible. (3) Override – “IAP” will have an impact on testcase-level data collectors. Since test execution will be in parallel, the start/end events marking the execution of a particular test might get interleaved with those of any other test that might be executing in parallel. Therefore make it possible for a feature that requires data collection to override and disable “IAP”.

“IAP” is OFF by default. It is intentionally left to the user to ensure that the tests are ready to be executed in parallel before enabling “IAP” – some kinds of tests, for e.g. functional tests, might not be suitable for “IAP”.

Approach The simplest way to enable “IAP” is globally for all MSTest V2 test assemblies using a .runsettings file as follows:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <!-- MSTest adapter -->
  <MSTest>
    <Parallelize>
      <Workers>4</Workers>
      <Scope>ClassLevel</Scope>
    </Parallelize>
  </MSTest>
</RunSettings>

This is as if every assembly were annotated with the following: [assembly: Parallelize(Workers = 4, Scope = ExecutionScope.ClassLevel)]

For each assembly then, parallel execution is realized by spawning the appropriate number of worker threads (4) on which to execute tests at the specified scope.

There are 3 scopes of parallelization: (1) ClassLevel – each thread executes a TestClass worth of tests. Within the TestClass, the test methods execute serially. This is the default – tests within a class might have interdependency, and we don’t want to be too aggressive. (2) MethodLevel – each thread executes a TestMethod. (3) Custom – the user can provide a plugin implementing the required execution semantics. This is presently not yet supported but mentioned because – like all of MSTest V2 – we have designed the feature with extensiblity in mind.

The number of worker threads to spawn can be set using the Workers parameter whose values can be as follows: 0 – Auto configure; use as many threads as possible based on CPU and core count. n – The number (integer) ‘n’ of threads to spawn to executes tests.

A TestAssembly, TestClass, TestMethod can explicitly opt-out of parallelization using the [DoNotParallelize()]) attribute. When used at the assembly level, all tests within the assembly will opt-out, and be executed serially. When used at the Class level, all tests within the class will opt-out, and be executed serially after the parallel execution of all other tests is completed. When used at the Method level, the test method will opt-out, and be executed serially after the parallel execution of all other tests is completed.

Finally, just as “IAP” can be enabled globally via the .runsettings file, it can be also be disabled globally as follows:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <!-- Configurations that affect the Test Framework -->
  <RunConfiguration>
    <DisableParallelization>true</DisableParallelization>
  </RunConfiguration>
</RunSettings>

Test lifecycle method semantics There is no change to the semantic of the Test lifecycle methods: – AssemblyInitialize/Cleanup shall be run only once per assembly. – ClassInitialize/Cleanup shall be run only once per class. – TestInitialize/Cleanup shall be run only once per method.

Conditioning “IAP” – settings waterfall As already described, “IAP” can be conditioned using the following means: (1) as annotations in source code. (2) as configuration properties via a .runsettings file. (3) by passing runsettings arguments via a command line.

(3) overrides (2) which in turn overrides (1). The [DoNotParallelize()]) annotation may be applied only to source code, and hence remains unaffected by these rules – thus, even if “IAP” in conditioned via (2) or (3), specific artifacts can opt-out safely.

Parallel Composition “Efficient Execution” is a key theme for the vstest platform, and here is how “IAP” composes over the vstest runner’s process parallelization, and VSTS VSTest task’s distributed test execution: The composition is along multiple axes: (1) “IAP”: this is at the Test framework/adapter layer. In vstest, the test Framework and its adapter work within the test host process. Thus, “IAP” is within a process for all adapters. Process level parallelization is supported by the vstest runner in the next layer. (2) Process-parallel execution: this is at the vstest runner layer, and supports parallelization across assemblies. It works by launching the test execution host on each available core, and handing it tests in an assembly to execute. This combined with (1) provides maximum degree of parallelization when tests run on 1 agent. (3) VSTS distributed test execution: this is at the VSTest task layer. The task supports distributing tests across multiple agents. Tests are grouped in batches and each agent executes 1 batch at a time. We support 3 different ways of batching: (3a) Based on number of tests and agents: Tests are grouped in equal sized batches. A batch could contain tests from one or more assemblies. Execution on the agent then conforms to (2) and (1). (3b) Based on past running time: Based on past test execution times and count of agents, tests are grouped into batches such that each batch takes approximately equal execution time. A batch could contain tests from one or more assemblies. Execution on the agent then conforms to (2) and (1). (3c) Based on assemblies: A test assembly is a batch. Thus, a batch would contain tests belonging to the same assembly. Execution on the agent then conforms to (2) and (1), but (2) is a no-op.

Summary and feedback So there you have it. We are eager to hear your feedback. You can provide it as comments on this post, or directly on issue on the testfx repo. We look forward to hearing from you.