Introducing the MSTest Runner – CLI, Visual Studio, & More

Amaury Levé

Marco Rossignoli

Jakub Jareš

January 24th, 202425 9

It is our pleasure to introduce MSTest runner, a new lightweight runner for MSTest tests. This new runner makes tests more portable and reliable, makes tests run faster and is extensible to provide you with an a la carte testing experience to add the tools you need to be successful.

What is it?

MSTest runner is a way to build and run MSTest tests as an independent portable executable. A simple console application is used to host and run your tests, so you don’t need any external tools such as vstest.console, dotnet test, or Visual Studio, to run your tests. Making this the perfect tool for authoring tests for devices with limited power or storage.

Installing MSTest runner

Developers of all experience levels and projects of any size can take advantage of the speed and portability of the new MSTest runner. We welcome you to try it out!

MSTest runner comes bundled with MSTest.TestAdapter NuGet package since version 3.2.0.

Enabling it for your project is as simple as installing the updated package and setting two MSBuild properties, <EnableMSTestRunner> and <OutputType>:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <!-- Enable the MSTest runner, this is an opt-in feature -->
    <EnableMSTestRunner>true</EnableMSTestRunner>
    <!-- We need to produce an executable and not a DLL -->
    <OutputType>Exe</OutputType>

    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>

    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <!-- 
      MSTest meta package is the recommended way to reference MSTest.
      It's equivalent to referencing:
          Microsoft.NET.Test.Sdk
          MSTest.TestAdapter
          MSTest.TestFramework
          MSTest.Analyzers
    -->    
    <PackageReference Include="MSTest" Version="3.2.0" />

  </ItemGroup>

</Project>

After making these changes, re-build your test project and your tests will create an executable that directly runs your tests:

Test summary showing 1 passed test.

Full example – Simple1

In the screenshot above you see that we did not need to run dotnet test, use vstest.console or run in Visual Studio to run our tests. Our tests are just a normal console application that discovers and runs tests.

That said the runner does integrate with dotnet test, vstest.console, Visual Studio Test Explorer and Visual Studio Code Test Explorer to provide you with the same experience you are used to. See our documentation to learn more.

Benefits of using the runner vs. VSTest

Portability

Running tests directly from an executable removes a lot of the complexity and infrastructure that is normally needed to run tests. Because test projects are no longer special, you can use the existing dotnet tooling to do interesting things with your test projects, such as building them as self-contained:

dotnet publish --runtime win-x64 --self-contained

The example above will publish the test project together with the runtime it needs to run. This allows you to move the project to a computer that does not have this runtime and run your tests on multiple computers without additional setup.

Or you can use this capability to create a zip file after every failed test run, to reproduce the failure locally the same way it failed on your CI server and get an easy way to debug your failed runs interactively.

Here is another example of running tests against a dotnet application hosted in a docker container that has no dotnet SDK available. A scenario that is a frequent stumbling point for our advanced users:

RunInDocker> docker build . -t my-server-tests

RunInDocker> docker run my-server-tests
Microsoft(R) Testing Platform Execution Command Line Tool
Version: 1.0.0-preview.23622.9+fe96e7475 (UTC 2023/12/22)
RuntimeInformation: linux-x64 - .NET 8.0.0
Copyright(c) Microsoft Corporation.  All rights reserved.
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://[::]:8080
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /test/test
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://localhost:8080/hello - - -
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'HTTP: GET /hello'
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'HTTP: GET /hello'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:8080/hello - 200 - text/plain;+charset=utf-8 73.5556ms
Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 1.7s - MyServer.Tests.dll (linux-x64 - .NET 8.0.0)

Full example – RunInDocker

Another advantage of MSTest runner portability is that you can now easily debug your tests as you would do for any regular executable. For example, in Visual Studio you can now simply:

  1. Navigate the test project you want to run in Solution Explorer, right select it and select Set as Startup Project.
  2. Navigate to the test you want to debug and add a breakpoint
  3. Select Debug > Start Debugging (or use F5) to run the selected test project.

You can also use --filter to filter down to the method or methods you want to debug to speed-up debugging experience. For example, --filter MSTestNamespace.UnitTest1.TestMethod2 to allow running (debugging) only the test method TestMethod2 from the class UnitTest1 in namespace MSTestNamespace. You can find more information about available filters at text. Here is an example of a launchSettings.json:

{
  "profiles": {
    "MSTestProject": {
      "commandName": "Project",
      "commandLineArgs": "--filter MSTestNamespace.UnitTest1.TestMethod2"
    }
  }
}

Finally, we are looking into making MSTest NativeAOT compatible, to let you test your applications in NativeAOT mode. To be able to do this we need to significantly change the internals of MSTest, please add a comment or thumbs up on our GitHub issue, if you find this useful.

Performance

MSTest runner uses one less process, and one less process-hop to run tests (when compared to dotnet test), to save resources on your build server.

It also avoids the need for inter-process serialized communication and relies on modern .NET APIs to increase parallelism and reduce footprint.

In the internal Microsoft projects that switched to use the new MSTest runner, we saw massive savings in both CPU and memory. Some projects seen were able to complete their tests 3 times as fast, while using 4 times less memory when running with dotnet test.

Even though those numbers might be impressive, there are much bigger gains to get when you enable parallel test runs in your test project. To help with this, we added a new set of analyzers for MSTest code analysis that promote good practice and correct setup of your tests.

Reliability

MSTest runner is setting new defaults, that are safer and make it much harder for you to accidentally miss running any of your tests. When making decisions we always err on the side of being stricter, and let you choose when you don’t need this strictness.

For example, MSTest runner will fail by default when there are zero tests run from a project, this can be controlled by --minimum-expected-tests, which defaults to 1. You can set it to 0, to not fail on you when there are no tests, but you can easily set it to a higher number to prevent regressions:

C:\p\testfx\samples\mstest-runner\Simple1> C:\p\testfx\artifacts\bin\Simple1\Debug\net8.0\Simple1.exe --minimum-expected-tests 10
Microsoft(R) Testing Platform Execution Command Line Tool
Version: 1.0.0-preview.23622.9+fe96e7475 (UTC 2023/12/22)
RuntimeInformation: win-x64 - .NET 8.0.0
Copyright(c) Microsoft Corporation.  All rights reserved.
Minimum expected tests policy violation, tests ran 1, minimum expected 10 - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 153ms - Simple1.dll (win-x64 - .NET 8.0.0)

But this is not the only reliability improvement. We wrote MSTest runner from ground up to make it more reliable.

MSTest runner, thanks to its new architecture, doesn’t rely on folder scanning, dynamic loading, or reflection to detect and load extensions. This makes it easier to have the same behavior on local and in CI, and it reduces the time between starting the test application and running the first test significantly.

The runner is designed to be async and parallelizable all the way, preventing some of the hangs or deadlocks that can be noticed when using VSTest.

The runner does not detect the target framework or the platform, or any other .NET configuration. It fully relies on the .NET platform to do that. This avoids duplication of logic, and avoids many edge cases that would break your tests when the rules suddenly change.

Extensibility

MSTest runner is based on a new barebone testing platform and an extensibility model that makes it easy to extend or override many aspects of the test execution.

It is now easy to provide your own report generator, test orchestration, loggers or even to increase the available command line options.

Microsoft is providing a list of optional extensions for you to be equipped with all you need to run and troubleshoot your tests.

We will continue to work on providing more extensions and features to enrich your testing experience. If you have specific needs or would like to help with growing the library extensions, please reach out to us.

Summary

MSTest runner is a performant, hostable, extensible, reliable, and integrated solution for running your MSTest tests. Whether you are a tech enthusiast, you are facing some issues with VSTest or simply curious, we welcome you to try it out and share your feedback below this article.

Special thanks

We would like to thank the team, whose relentless efforts and unwavering commitment brought this feature to fruition.

Additionally, we would like to express our heartfelt gratitude to the internal teams who helped dogfood and support this initiative.

Amaury Levé Senior Software Engineer, .NET Developer Experience

Marco Rossignoli Senior Software Engineer, DevDiv - .NET Developer Experience

Jakub Jareš Senior Software Engineer, Testing Platform

Read next

25 comments

Discussion is closed. Login to edit/delete existing comments.

  • Michael Taylor 2

    Is this going to be the new recommendation going forward? In other words is the existing MSTest project template going to be updated to use these settings?

    What happens with Test Explorer in VS? Based upon the article it seems like it remains unchanged but what about extensions that hook into that service and provide more functionality (e.g. CodeRush)?

    What about build pipelines that use VS Test or dotnet CLI to run tests? Will they still work with a project configured this way or will they fail to run? I’m wondering about cases where there are multiple unit test projects in a solution and only some of them are upgraded.

    • Jakub JarešMicrosoft employee 2

      Hi Michael,
      Thank you for your interest and for all these questions!

      Is this going to be the new recommendation going forward? In other words is the existing MSTest project template going to be updated to use these settings?

      Yes, we recommend using MSTest runner for all new MSTest projects. We will soon update MSTest templates to use MSTest runner by default, and will provide a parameter to opt-out, such as dotnet new mstest --disable-runner.

      What happens with Test Explorer in VS? Based upon the article it seems like it remains unchanged but what about extensions that hook into that service and provide more functionality (e.g. CodeRush)?

      We took a lot of care to stay compatible with the current ways of running tests. We kept Test Explorer working as is, even though we implemented a new open protocol to talk with Test Explorer in Visual Studio and Visual Studio Code. We are still documenting this and will have it ready in time for VS17.10.

      We also kept compatibility with dotnet test, and as opt-in we are adding a new streamlined mode for dotnet test, please refer to this article this article for more information on how to enable it.

      What about build pipelines that use VS Test or dotnet CLI to run tests? Will they still work with a project configured this way or will they fail to run? I’m wondering about cases where there are multiple unit test projects in a solution and only some of them are upgraded.

      We are also keeping these scenarios backwards compatible, your tests should run as they did until now, with the added benefit of being able to run the test project directly in command line as an exe. If you decide to opt-in for the new dotnet-test or new Test Explorer experience you will be able to use the new features, but some existing scenarios might break.

      For solutions that are partially migrated to MSTest runner, you should not see problems running tests through the current dotnet test. If you opt-into the new dotnet test experience in your projects, you might need to consider the legacy, and opted-in projects as two different runs. They will need different parameters, and will return different attachments etc.

  • Michael Dietrich 2

    This article always refers to MSTest and does not mention any other test frameworks at all (e.g. xUnit, NUnit, ..).
    Does it mean using such frameworks just works with the new approach without needing to switch away from these frameworks or does it mean these frameworks are not (yet) supported?
    Could you please clarify this?

    • Jakub JarešMicrosoft employee 3

      Yes this is true. The runner is only for MSTest (for now), but we’ve built it on building blocks that are framework agnostic, and then after a series of discussions went with just MSTest. This gives us a better flexibility, and easier time providing a solution that is backwards compatible. Because the scope is smaller and focused on just one framework, we can do changes quicker, and with more confidence, because we are most familiar with MSTest codebase.

      Please comment and vote here if you’d like this for other frameworks:

      https://github.com/microsoft/testfx/issues/2164

  • Michael Dietrich 3

    Could you please be more clear on the benefits of this new approach? E.g. does it improve performance or so?

    I like the idea of making test projects actual executables, but I do not really get the point about the better portability and so.

    To be more clear, in our case we already separate building and executing tests using different machines, the only dependency for executing tests on another machine is having the .NET SDK installed, which is fairly easy to achieve with dotnet-install-script while publishing test projects as self-contained can result in a huge amount of data depending on the number of test projects, not to mention that publishing can take a fair while for some projects especially with using AOT (e.g. tests for Blazor projects).

    So, maybe you can give some more insights about what this new approach can bring to developers and DevOps in general and also have in mind to let existing workflows to still work in the future. 🙂

    • Jakub JarešMicrosoft employee 5

      Sure.

      We want to better align with the rest of the dotnet sdk and other dotnet tools. There have been huge strides in what dotnet sdk can do, like trimming, native aot, multi targeting and so on. Each of those changes require a lot of work on vstest side to allow this experience and keep up with it, because vstest is a central place that needs to know how to run any tests, both old and new. By moving to tests being executables, we need to know a lot less about these more intricate modes of running dotnet apps, and can offer a more timely experience to the users who need it. We don’t want to force you to run as self-contained, and make your test projects bigger if you don’t have the need. But if you have that need, or can see benefit from running without SDK we want to allow it.

      In similar manner we would like to allow tests to be truly portable, where you can reproduce the same issues locally as on your build server, with ease. Currently you have to consider many things, such as version of VS that is installed on the target machine or version of dotnet SDK, additional extension lookup paths, and so on. In our ideal world you end up with a directory that can be zipped on test failure, uploaded as artifact, and will repro the issue on your own workstation, together with all necessary symbols for interactive debugging.

      Another category of improvements are performance. With vstest all test runs need to go through vstest.console. When running simple test suite without data collectors, this adds unnecessary overhead. By moving the runner (the equivalent of vstest.console) in process we can avoid serialization for inter process communication, and additional memory consumption for deserialization. We are also looking into improvements that we can do to the mstest framework itself, which is our current bottleneck at running tests with lower overhead. But even in the current state we are seeing huge gains (or better said reduction in overhead ), when comparing projects that run with vstest and with mstest runner: https://github.com/microsoft/testfx/tree/main/samples/runner_vs_vstest#results-of-comparison

      We also shorten our feedback and release loop to minimum. Because we only ship as nuget packages, we can (in theory) release every day, and ship fixes the next day. This unties our hands when doing bigger changes, such as moving the whole stack to be async await compatible. (This will not mean that we will break you every day, we are trying hard to keep the current workflows working, while enabling more workflows to be possible.)

      As with all our changes, additional feedback is definitely welcome, and you can file it on https://github.com/microsoft/testfx any time.

      • Michael Dietrich 3

        Thank you for these details.
        Sounds awesome, you should’ve added an outlook section to your blog post to get people to be even more interested. 😀

  • Joey Xie 1

    I found the new MSTest:3.2.0 with outputtype exe is not compatible with vscode test explorer, no test shows.

    • Jakub JarešMicrosoft employee 1

      Thanks, we will investigate this.

        • Sam Illingworth 0

          I followed your link https://github.com/microsoft/testfx/issues/2171 link. But, the issue with the MSTest:3.2.0 with outputtype exe not being compatible with vscode test explorer still remains.

          • Amaury LevéMicrosoft employee 0

            Hi Sam,

            We just released MSTest 3.2.1 with a bug fix that was causing the issue with VSCode. Our team has confirmed the issue is fixed, would you mind giving it a new try?

  • Eric Lamontagne 1

    Hello and thank you for the improvement!

    Our biggest project is still using .Net 4.7.2, I’m not sure if this is supported?

    Regards,

    Éric

    • Amaury LevéMicrosoft employee 0

      We follow supported frameworks so down to net462.

      Looking forward for your feedback 🙂

  • Luca C. 1

    Is it possible to have in console the standard captured output, as in Test Explorer? Example…

    Multiline Data To Json
     Source: MultilineToJsonTests.cs line 13
     Duration: 89 ms

    Standard Output: 

    TestContext Messages:
    Information: Multiline data to json start

    Debug: Extension Raven.Extensions.MultilineDataToJson [v. 1.0.0] keep alive [execution start date = 09:28:36.600; status = Running; completion perc = 0]

    Information: Multiline data to json end

    • Jakub JarešMicrosoft employee 2

      Thank you for the feedback.

      We are looking into what we are printing, in the first release we went with the minimum. We are revisiting this, based on your and others feedback. There are a lot of things to consider, and balance. Some users run tests in parallel, and would like output at the end, some would like to have it right away because they run long-running serial tests, some need native output, some don’t. But technically nothing is preventing us from printing this information, MSTest is doing all the heavy lifting here, and just passes that information to VSTest that passes it to TestExplorer.

      https://github.com/microsoft/testfx/issues/2162
      https://github.com/microsoft/testfx/issues/2172

  • Rafsanul Hasan 1

    This is really promising. MSTest will be next vital thing which could potentially replace NUnit/xUnit. But as a mocking library, I really love Microsoft Fakes which priorly used to require VSTest. Can we use Microsoft Fakes with MSTest Framework?

    • Amaury LevéMicrosoft employee 1

      Hi Rafsanul,

      I will need to double check with the team but I think there are so modifications required to enable MS Fakes to work with MSTest runner. I have created this issue for you to track: https://github.com/microsoft/testfx/issues/2186

  • Micah Rairdon 0

    We’re using VSTest to run our E2E tests which by their nature are very flakey. As of now the only way to retry automatically is to use the VSTest task in Azure DevOps.

    Are you planning on bringing a retry feature to this adapter?

    • Amaury LevéMicrosoft employee 0

      Hi Micah,

      We already do 🙂 checkout the extension page of MSTest runner. This retry capability allows you to rerun failed tests locally and on Linux and Mac which wasn’t possible with VSTest task.

      Please do pay attention to the licensing model of this extension.

      • Joey Gerits 0

        unfortunately the retry extensions gives errors when running. We get: This program is thought to be unreachable. File=’/_/src/Microsoft.Testing.Extensions.Retry/RetryOrchestrator.cs’ Line 73.
        What is the best place to report this issue because i don’t think this should be in the testfx github repository.

  • Jack DeMeyers 0

    I’m wondering if the following scenario is a supported situation.

    We have an Azure Function app, and an EndToEnd test project. Generally, we point the E2E tests at our deployed environment in Azure. However, sometimes if we are experiencing test failures in our Azure environment, it can be useful to debug both the Tests and the Function app locally at the same time.

    Prior to the new runner, the process was:
    1. start the function app without debugging (ctrl-F5)
    2. add a breakpoint at the start of the test
    3. start debugging the test
    4. once breakpoint in test is hit, manually attach debugger to the function app process

    Then you can hit break points in both the functions and the tests. However, this isn’t an ideal solution due to the need to manually attach the debugger.

    With the new runner, it appears the following process works, and is much easier:

    1. configure multiple startup projects
      • setup test project and function app to start
    2. start the projects as normal (run button in Visual Studio)

    Example/POC repo – https://github.com/jkdmyrs/New-MSTestRunner-Experiment

    But I’m wondering if this is a supported use case.
    Are we always garunteed that the function app will start up before the tests run?
    Or are there race-conditions where the tests exectute before the function HTTP endpoints are available?

  • Abdessattar Sassi 0

    Is this runner supposed to be compatible with test projects that use “UITestMethod”? or test projects that generally need the WindowsAppSdk?

    If yes, could you please explain what needs to be changed?

Feedback usabilla icon