It is our pleasure to introduce MSTest runner, a new lightweight runner for MSTest tests. This new runner makes tests more portable and reliable, makes tests run faster and is extensible to provide you with an a la carte testing experience to add the tools you need to be successful.
What is it?
MSTest runner is a way to build and run MSTest tests as an independent portable executable. A simple console application is used to host and run your tests, so you don’t need any external tools such as vstest.console, dotnet test, or Visual Studio, to run your tests. Making this the perfect tool for authoring tests for devices with limited power or storage.
Installing MSTest runner
Developers of all experience levels and projects of any size can take advantage of the speed and portability of the new MSTest runner. We welcome you to try it out!
MSTest runner comes bundled with MSTest.TestAdapter NuGet package since version 3.2.0.
Enabling it for your project is as simple as installing the updated package and setting two MSBuild properties, <EnableMSTestRunner> and <OutputType>:
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<!-- Enable the MSTest runner, this is an opt-in feature -->
<EnableMSTestRunner>true</EnableMSTestRunner>
<!-- We need to produce an executable and not a DLL -->
<OutputType>Exe</OutputType>
<TargetFramework>net8.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<IsPackable>false</IsPackable>
</PropertyGroup>
<ItemGroup>
<!--
MSTest meta package is the recommended way to reference MSTest.
It's equivalent to referencing:
Microsoft.NET.Test.Sdk
MSTest.TestAdapter
MSTest.TestFramework
MSTest.Analyzers
-->
<PackageReference Include="MSTest" Version="3.2.0" />
</ItemGroup>
</Project>After making these changes, re-build your test project and your tests will create an executable that directly runs your tests:
In the screenshot above you see that we did not need to run dotnet test, use vstest.console or run in Visual Studio to run our tests. Our tests are just a normal console application that discovers and runs tests.
That said the runner does integrate with dotnet test, vstest.console, Visual Studio Test Explorer and Visual Studio Code Test Explorer to provide you with the same experience you are used to. See our documentation to learn more.
Benefits of using the runner vs. VSTest
Portability
Running tests directly from an executable removes a lot of the complexity and infrastructure that is normally needed to run tests. Because test projects are no longer special, you can use the existing dotnet tooling to do interesting things with your test projects, such as building them as self-contained:
dotnet publish --runtime win-x64 --self-containedThe example above will publish the test project together with the runtime it needs to run. This allows you to move the project to a computer that does not have this runtime and run your tests on multiple computers without additional setup.
Or you can use this capability to create a zip file after every failed test run, to reproduce the failure locally the same way it failed on your CI server and get an easy way to debug your failed runs interactively.
Here is another example of running tests against a dotnet application hosted in a docker container that has no dotnet SDK available. A scenario that is a frequent stumbling point for our advanced users:
RunInDocker> docker build . -t my-server-tests
RunInDocker> docker run my-server-tests
Microsoft(R) Testing Platform Execution Command Line Tool
Version: 1.0.0-preview.23622.9+fe96e7475 (UTC 2023/12/22)
RuntimeInformation: linux-x64 - .NET 8.0.0
Copyright(c) Microsoft Corporation. All rights reserved.
info: Microsoft.Hosting.Lifetime[14]
Now listening on: http://[::]:8080
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
Content root path: /test/test
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:8080/hello - - -
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
Executing endpoint 'HTTP: GET /hello'
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
Executed endpoint 'HTTP: GET /hello'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished HTTP/1.1 GET http://localhost:8080/hello - 200 - text/plain;+charset=utf-8 73.5556ms
Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 1.7s - MyServer.Tests.dll (linux-x64 - .NET 8.0.0)Another advantage of MSTest runner portability is that you can now easily debug your tests as you would do for any regular executable. For example, in Visual Studio you can now simply:
- Navigate the test project you want to run in Solution Explorer, right select it and select Set as Startup Project.
- Navigate to the test you want to debug and add a breakpoint
- Select Debug > Start Debugging (or use F5) to run the selected test project.
You can also use --filter to filter down to the method or methods you want to debug to speed-up debugging experience. For example, --filter MSTestNamespace.UnitTest1.TestMethod2 to allow running (debugging) only the test method TestMethod2 from the class UnitTest1 in namespace MSTestNamespace. You can find more information about available filters at text.
Here is an example of a launchSettings.json:
{
"profiles": {
"MSTestProject": {
"commandName": "Project",
"commandLineArgs": "--filter MSTestNamespace.UnitTest1.TestMethod2"
}
}
}Finally, we are looking into making MSTest NativeAOT compatible, to let you test your applications in NativeAOT mode. To be able to do this we need to significantly change the internals of MSTest, please add a comment or thumbs up on our GitHub issue, if you find this useful.
Performance
MSTest runner uses one less process, and one less process-hop to run tests (when compared to dotnet test), to save resources on your build server.
It also avoids the need for inter-process serialized communication and relies on modern .NET APIs to increase parallelism and reduce footprint.
In the internal Microsoft projects that switched to use the new MSTest runner, we saw massive savings in both CPU and memory. Some projects seen were able to complete their tests 3 times as fast, while using 4 times less memory when running with dotnet test.
Even though those numbers might be impressive, there are much bigger gains to get when you enable parallel test runs in your test project. To help with this, we added a new set of analyzers for MSTest code analysis that promote good practice and correct setup of your tests.
Reliability
MSTest runner is setting new defaults, that are safer and make it much harder for you to accidentally miss running any of your tests. When making decisions we always err on the side of being stricter, and let you choose when you don’t need this strictness.
For example, MSTest runner will fail by default when there are zero tests run from a project, this can be controlled by --minimum-expected-tests, which defaults to 1, but you can easily set it to a higher number to prevent regressions:
C:\p\testfx\samples\mstest-runner\Simple1> C:\p\testfx\artifacts\bin\Simple1\Debug\net8.0\Simple1.exe --minimum-expected-tests 10
Microsoft(R) Testing Platform Execution Command Line Tool
Version: 1.0.0-preview.23622.9+fe96e7475 (UTC 2023/12/22)
RuntimeInformation: win-x64 - .NET 8.0.0
Copyright(c) Microsoft Corporation. All rights reserved.
Minimum expected tests policy violation, tests ran 1, minimum expected 10 - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 153ms - Simple1.dll (win-x64 - .NET 8.0.0)To not fail on you when there are no tests, you can use --ignore-exit-code 8, learn more about how to ignore some special exit codes.
But this is not the only reliability improvement. We wrote MSTest runner from ground up to make it more reliable.
MSTest runner, thanks to its new architecture, doesn’t rely on folder scanning, dynamic loading, or reflection to detect and load extensions. This makes it easier to have the same behavior on local and in CI, and it reduces the time between starting the test application and running the first test significantly.
The runner is designed to be async and parallelizable all the way, preventing some of the hangs or deadlocks that can be noticed when using VSTest.
The runner does not detect the target framework or the platform, or any other .NET configuration. It fully relies on the .NET platform to do that. This avoids duplication of logic, and avoids many edge cases that would break your tests when the rules suddenly change.
Extensibility
MSTest runner is based on a new barebone testing platform and an extensibility model that makes it easy to extend or override many aspects of the test execution.
It is now easy to provide your own report generator, test orchestration, loggers or even to increase the available command line options.
Microsoft is providing a list of optional extensions for you to be equipped with all you need to run and troubleshoot your tests.
We will continue to work on providing more extensions and features to enrich your testing experience. If you have specific needs or would like to help with growing the library extensions, please reach out to us.
Summary
MSTest runner is a performant, hostable, extensible, reliable, and integrated solution for running your MSTest tests. Whether you are a tech enthusiast, you are facing some issues with VSTest or simply curious, we welcome you to try it out and share your feedback below this article.
Special thanks
We would like to thank the team, whose relentless efforts and unwavering commitment brought this feature to fruition.
Additionally, we would like to express our heartfelt gratitude to the internal teams who helped dogfood and support this initiative.
Is this runner supposed to be compatible with test projects that use “UITestMethod”? or test projects that generally need the WindowsAppSdk?
If yes, could you please explain what needs to be changed?
I'm wondering if the following scenario is a supported situation.
We have an Azure Function app, and an EndToEnd test project. Generally, we point the E2E tests at our deployed environment in Azure. However, sometimes if we are experiencing test failures in our Azure environment, it can be useful to debug both the Tests and the Function app locally at the same time.
Prior to the new runner, the process was:
1. start the function app without...
Hi Jack,
also if we could have the guarantee on the startup order we cannot know if the function is ready to accept the connection.
I’ve explained how you can achieve your workflow synchronizing the two console here https://github.com/jkdmyrs/New-MSTestRunner-Experiment/pull/1
We’re using VSTest to run our E2E tests which by their nature are very flakey. As of now the only way to retry automatically is to use the VSTest task in Azure DevOps.
Are you planning on bringing a retry feature to this adapter?
Hi Micah,
We already do 🙂 checkout the extension page of MSTest runner. This retry capability allows you to rerun failed tests locally and on Linux and Mac which wasn’t possible with VSTest task.
Please do pay attention to the licensing model of this extension.
unfortunately the retry extensions gives errors when running. We get: This program is thought to be unreachable. File=’/_/src/Microsoft.Testing.Extensions.Retry/RetryOrchestrator.cs’ Line 73.
What is the best place to report this issue because i don’t think this should be in the testfx github repository.
TestFx is the best place to report it, made one on your behalf: https://github.com/microsoft/testfx/issues/2189
This is really promising. MSTest will be next vital thing which could potentially replace NUnit/xUnit. But as a mocking library, I really love Microsoft Fakes which priorly used to require VSTest. Can we use Microsoft Fakes with MSTest Framework?
Hi Rafsanul,
I will need to double check with the team but I think there are so modifications required to enable MS Fakes to work with MSTest runner. I have created this issue for you to track: https://github.com/microsoft/testfx/issues/2186
Is it possible to have in console the standard captured output, as in Test Explorer? Example…
Multiline Data To Json
Source: MultilineToJsonTests.cs line 13
Duration: 89 ms
Standard Output:
TestContext Messages:
Information: Multiline data to json start
Debug: Extension Raven.Extensions.MultilineDataToJson [v. 1.0.0] keep alive [execution start date = 09:28:36.600; status = Running; completion perc = 0]
Information: Multiline data to json end
Thank you for the feedback.
We are looking into what we are printing, in the first release we went with the minimum. We are revisiting this, based on your and others feedback. There are a lot of things to consider, and balance. Some users run tests in parallel, and would like output at the end, some would like to have it right away because they run long-running serial tests, some need native output, some don't. But...
Hello and thank you for the improvement!
Our biggest project is still using .Net 4.7.2, I’m not sure if this is supported?
Regards,
Éric
We follow supported frameworks so down to net462.
Looking forward for your feedback 🙂
I found the new
MSTest:3.2.0with outputtype exe is not compatible with vscode test explorer, no test shows.Thanks, we will investigate this.
You can follow https://github.com/microsoft/testfx/issues/2171
I followed your link https://github.com/microsoft/testfx/issues/2171 link. But, the issue with the MSTest:3.2.0 with outputtype exe not being compatible with vscode test explorer still remains.
Hi Sam,
We just released MSTest 3.2.1 with a bug fix that was causing the issue with VSCode. Our team has confirmed the issue is fixed, would you mind giving it a new try?
Could you please be more clear on the benefits of this new approach? E.g. does it improve performance or so?
I like the idea of making test projects actual executables, but I do not really get the point about the better portability and so.
To be more clear, in our case we already separate building and executing tests using different machines, the only dependency for executing tests on another machine is having the .NET SDK installed, which...
Sure.
We want to better align with the rest of the dotnet sdk and other dotnet tools. There have been huge strides in what dotnet sdk can do, like trimming, native aot, multi targeting and so on. Each of those changes require a lot of work on vstest side to allow this experience and keep up with it, because vstest is a central place that needs to know how to run any tests, both old and...
Thank you for these details.
Sounds awesome, you should’ve added an outlook section to your blog post to get people to be even more interested. 😀
This article always refers to
MSTestand does not mention any other test frameworks at all (e.g. xUnit, NUnit, ..).Does it mean using such frameworks just works with the new approach without needing to switch away from these frameworks or does it mean these frameworks are not (yet) supported?
Could you please clarify this?
Yes this is true. The runner is only for MSTest (for now), but we've built it on building blocks that are framework agnostic, and then after a series of discussions went with just MSTest. This gives us a better flexibility, and easier time providing a solution that is backwards compatible. Because the scope is smaller and focused on just one framework, we can do changes quicker, and with more confidence, because we are most familiar...
Is this going to be the new recommendation going forward? In other words is the existing MSTest project template going to be updated to use these settings?
What happens with Test Explorer in VS? Based upon the article it seems like it remains unchanged but what about extensions that hook into that service and provide more functionality (e.g. CodeRush)?
What about build pipelines that use VS Test or dotnet CLI to run tests? Will they still work...
Hi Michael,
Thank you for your interest and for all these questions!
Is this going to be the new recommendation going forward? In other words is the existing MSTest project template going to be updated to use these settings?
Yes, we recommend using MSTest runner for all new MSTest projects. We will soon update MSTest templates to use MSTest runner by default, and will provide a parameter to opt-out, such as .
What happens...