Test batching options with VSTest task
Being able to run tests efficiently in a CI/CD pipeline is crucial to get a fast indication of build quality in order to quickly deliver customer value. For this, VSTest supports running automated tests in a distributed manner using multiple agents. In this post, we will discuss the various options available in the VSTest task to distribute tests and the suitability of each option.
Option 1: Based on number of tests and agents
In this case, the tests are grouped into batches of equal count of tests. By default, the number of tests per batch is determined as follows:
Count of tests per batch = (Total no. of tests / No. of agents)
You can also specify your own batch size by selecting the “Specify a batch size” option.
This option gives you the flexibility to decide the number of tests to run in a batch. This will work best for you when you have tests of approximately equal duration. However, when the tests have widely varying execution times, the distribution may not be uniform in terms of the time taken to execute the tests in each agent. Some agents may end up getting tests of longer duration, while others get an equal number of tests but of much lesser duration. This results in some agents finishing early and then waiting idly for the other agents to complete their tests. To overcome this, you may consider the next option described below.
Option 2: Based on past running time of tests
In this case, the tests are grouped into batches of approximately equal execution times, where the execution time of the tests is determined from the latest previous build/release that executed any test. By default, the running time per batch is automatically determined using the below heuristic based on an acceptable batching overhead:
Running time per batch = 10 % of (Sum of past execution times of all test cases /No. of agents)
You can also specify your own batch time using the “Specify running time per batch” option. The tests will then be grouped into batches of approximately the specified running time.
This option gives you the flexibility to decide how long each batch should run, thereby also ensuring an efficient utilization of agents. This will work best for you when you have tests of widely varying duration.
Some interesting considerations when using Option 2 “Based on past running time of tests”
- If the previous run did not execute some tests that are getting executed in the current run, then the execution time of these tests will be considered to be unknown and such tests will be distributed evenly among the agents, based on count. For example, if you have 3 agents and build N ran 10 tests and build N+1 runs 16 tests (including 6 new tests), apart from the batches created for the 10 old tests based on the past execution time, there will be 3 more batches each containing 2 new test cases.
- If some tests failed in the previous run, then the execution time of these tests will also be considered to be unknown and such tests will be distributed evenly among the agents based on count.
- If there are test cases whose execution time is greater that the running time per batch, they are each placed in a batch of their own.
- If the previous build/release had multiple VSTest tasks, and only one ran, that build will still be used to determine the execution times of tests. The execution time of the tests that did not run will be considered to be unknown and such tests will be distributed evenly among the agents, based on count.
- When test impact is used, it is very likely that the tests in the current run did not execute in the previous run. In such cases, the execution time of such tests will be considered to be unknown and they will be distributed evenly among the agents, based on count. For example, if you have 3 agents and if build N ran 30 tests and build N+1 runs 25 tests, 10 of which ran in build N and 15 that did not, the 10 tests will get distributed based on past execution time, but the 15 will randomly get distributed into batches of 5 each, agnostic of execution time.
Option 3: Based on test assemblies
In this case, the tests are grouped into as many batches as the number of test assemblies, with the tests belonging to the same assembly grouped into the same batch.
When using options 1 and 2, you might notice an increase in the individual test method execution times. This can happen if you have test methods with a ‘class initialize’ or ‘assembly initialize’ code. If these test cases end up in different batches, then the initialization will get executed in each of these batches, adding up to the test case execution time. To avoid this, you may choose to run all the tests belonging to the same assembly together by selecting this option.
The batching options discussed above are already available in VSTS.
The first two batching options discussed above are available from TFS 2018 onwards. The third distribution option “Based on test assemblies” will be available with a future update.
The multi-agent phase option is available only in the release pipeline right now, and is expected in the build pipeline soon. You can still use the above batching options in the build pipeline to run your tests in batches within a single agent. This will let you publish the results in batches, even though the batches will not run in parallel to reduce the overall test run time.
Reach out to devops_tools[at]microsoft[dot]com if you have further queries regarding this feature.