{"id":59196,"date":"2025-12-16T10:05:00","date_gmt":"2025-12-16T18:05:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/dotnet\/?p=59196"},"modified":"2025-12-16T10:05:00","modified_gmt":"2025-12-16T18:05:00","slug":"microsoft-testing-platform-azure-retry","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/dotnet\/microsoft-testing-platform-azure-retry\/","title":{"rendered":"Microsoft.Testing.Platform Now Fully Supported in Azure DevOps"},"content":{"rendered":"<p>Earlier this year, we announced that <a href=\"https:\/\/devblogs.microsoft.com\/dotnet\/mtp-adoption-frameworks\/\">all major .NET test frameworks now support Microsoft.Testing.Platform<\/a>. The next logical step? Making sure it works seamlessly with your CI\/CD pipelines. Today, we&#8217;re announcing comprehensive Azure DevOps integration for Microsoft.Testing.Platform.<\/p>\n<h2>What&#8217;s New in Azure DevOps<\/h2>\n<p>Azure DevOps now provides first-class support for Microsoft.Testing.Platform with two key improvements:<\/p>\n<ul>\n<li><strong>Run tests<\/strong> using the familiar DotNetCoreCLI task; no more workarounds required.<\/li>\n<li><strong>Handle test retries<\/strong> intelligently\u2014publish multiple TRX files from retry attempts with proper grouping and exit codes.<\/li>\n<\/ul>\n<p>Whether you&#8217;re migrating from VSTest or starting fresh with Microsoft.Testing.Platform, the experience is now smooth and intuitive.<\/p>\n<h2>About Microsoft.Testing.Platform<\/h2>\n<p>If you&#8217;re new to Microsoft.Testing.Platform, here are some helpful resources to get you up to speed:<\/p>\n<ul>\n<li><a href=\"https:\/\/learn.microsoft.com\/dotnet\/core\/testing\/\">Testing in .NET<\/a> gives an overview of the testing tools used in .NET. In particular, it clarifies the difference between a test platform and a test framework.<\/li>\n<li><a href=\"https:\/\/learn.microsoft.com\/dotnet\/core\/testing\/unit-testing-with-dotnet-test\">Testing with &#8216;dotnet test&#8217;<\/a> explains the different modes that the &#8216;dotnet test&#8217; command can operate in.<\/li>\n<li><a href=\"https:\/\/learn.microsoft.com\/dotnet\/core\/testing\/microsoft-testing-platform-intro\">Microsoft.Testing.Platform overview<\/a> covers what Microsoft.Testing.Platform is and how to run tests with it.<\/li>\n<li><a href=\"https:\/\/devblogs.microsoft.com\/dotnet\/dotnet-test-with-mtp\/\">Enhance your CLI testing workflow with the new dotnet test<\/a> shows the &#8216;dotnet test&#8217; support for Microsoft.Testing.Platform that was added in .NET 10 SDK.<\/li>\n<\/ul>\n<p>In short, Microsoft.Testing.Platform is a modern alternative to VSTest.<\/p>\n<h2>Running Tests in Azure DevOps<\/h2>\n<p>The DotNetCoreCLI task now supports Microsoft.Testing.Platform starting in version 2.263.0!<\/p>\n<p>You have two main options for running tests with Microsoft.Testing.Platform in Azure DevOps:<\/p>\n<h3>Option 1: Use the DotNetCoreCLI task (Recommended)<\/h3>\n<p>The <a href=\"https:\/\/learn.microsoft.com\/azure\/devops\/pipelines\/tasks\/reference\/dotnet-core-cli-v2\">DotNetCoreCLI Azure DevOps task<\/a> now <a href=\"https:\/\/github.com\/microsoft\/azure-pipelines-tasks\/pull\/21315\/\">supports Microsoft.Testing.Platform<\/a> starting in version 2.263.0.<\/p>\n<pre><code class=\"language-yml\">- task: DotNetCoreCLI@2\n  displayName: 'Run tests'\n  inputs:\n    command: 'test'\n    projects: '**\/*Tests.csproj'\n    arguments: '--no-build --report-trx'<\/code><\/pre>\n<p><strong>What changed?<\/strong> Microsoft.Testing.Platform uses different command-line flags than VSTest. For example, use <code>--report-trx<\/code> instead of <code>--logger trx<\/code>. See the <a href=\"https:\/\/learn.microsoft.com\/dotnet\/core\/testing\/microsoft-testing-platform-intro\">Microsoft.Testing.Platform CLI documentation<\/a> for the full list of options.<\/p>\n<h3>Option 2: Run &#8216;dotnet test&#8217; directly<\/h3>\n<p>You can run the <code>dotnet test<\/code> command directly using a script or command-line task. This gives you maximum flexibility and doesn&#8217;t require specific Azure DevOps task updates.<\/p>\n<pre><code class=\"language-yml\">- task: CmdLine@2\n  displayName: 'Run tests'\n  inputs:\n    script: 'dotnet test --no-build --report-trx'<\/code><\/pre>\n<p><strong>When to use this?<\/strong> Choose this approach if you need custom scripting around test execution or want to avoid dependencies on specific task versions.<\/p>\n<h3>Migrating from VSTest task<\/h3>\n<p>The <a href=\"https:\/\/learn.microsoft.com\/azure\/devops\/pipelines\/tasks\/reference\/vstest-v3\">VSTest Azure DevOps task<\/a> was designed exclusively for VSTest and will not support Microsoft.Testing.Platform.<\/p>\n<p><strong>If you&#8217;re currently using the VSTest task:<\/strong><\/p>\n<ol>\n<li>Switch to Option 1 (DotNetCoreCLI task) for the most similar experience<\/li>\n<li>Update your command-line arguments to use Microsoft.Testing.Platform syntax<\/li>\n<li>Test your pipeline to ensure test discovery and execution work as expected<\/li>\n<\/ol>\n<p>Both Option 1 and Option 2 provide the same test execution capabilities as the VSTest task.<\/p>\n<h2>Publishing test results with retry support<\/h2>\n<p>Here&#8217;s the challenge: when you use the <a href=\"https:\/\/www.nuget.org\/packages\/Microsoft.Testing.Extensions.Retry\">Retry extension<\/a> to automatically re-run failed tests, Microsoft.Testing.Platform generates a separate TRX file for each attempt. Previously, the <code>PublishTestResults<\/code> task treated these as independent test runs, leading to two problems:<\/p>\n<ul>\n<li><strong>Incorrect exit codes<\/strong>: The task would fail even when tests eventually passed after retry<\/li>\n<li><strong>Confusing UI<\/strong>: Test results appeared as separate runs instead of grouped retry attempts<\/li>\n<\/ul>\n<p>We&#8217;ve solved this! The <code>PublishTestResults<\/code> task now intelligently handles multiple TRX files from retry attempts, correctly grouping them and setting appropriate exit codes.<\/p>\n<h3>What you need to know<\/h3>\n<p>Microsoft.Testing.Platform uses the standard TRX format (also known as VSTest test results format) through the <a href=\"https:\/\/nuget.org\/packages\/Microsoft.Testing.Extensions.TrxReport\">Microsoft.Testing.Extensions.TrxReport<\/a> package. The <a href=\"https:\/\/learn.microsoft.com\/azure\/devops\/pipelines\/tasks\/reference\/publish-test-results-v2\">PublishTestResults task<\/a> has always supported TRX files. What&#8217;s new is the intelligent handling of retry scenarios.<\/p>\n<p><div class=\"alert alert-info\"><p class=\"alert-divider\"><i class=\"fabric-icon fabric-icon--Info\"><\/i><strong>Required Configuration<\/strong><\/p>To enable retry-aware test result publishing, set the <code>AllowPtrToDetectTestRunRetryFiles<\/code> variable to <code>true<\/code> in your pipeline. This opt-in flag activates the new behavior that correctly interprets multiple TRX files as retry attempts rather than separate test runs.<\/p>\n<p><strong>Important:<\/strong> This variable must be set at the pipeline level (in your YAML file or pipeline variables). It will not work if set at the project or organization level.<\/div><\/p>\n<h3>Complete pipeline example<\/h3>\n<p>Here&#8217;s a full pipeline demonstrating test execution with retries and proper result publishing:<\/p>\n<pre><code class=\"language-yml\">trigger:\n- main\n\njobs:\n- job: MyJob\n  variables:\n    AllowPtrToDetectTestRunRetryFiles: true\n\n  steps:\n\n  # Your usual steps to install .NET SDK, restore, and build.\n\n  - task: CmdLine@2\n    displayName: 'Run tests'\n    inputs:\n      script: 'dotnet test --no-build --report-trx --retry-failed-tests 3 --results-directory TestResults'\n\n  - task: PublishTestResults@2\n    displayName: 'Publish test results'\n    inputs:\n      testResultsFormat: 'VSTest'\n      testResultsFiles: '**\/*.trx'\n      mergeTestResults: true\n      failTaskOnFailedTests: true\n    condition: succeededOrFailed()<\/code><\/pre>\n<p><strong>Key points in this pipeline:<\/strong><\/p>\n<ul>\n<li>The <code>AllowPtrToDetectTestRunRetryFiles: true<\/code> variable enables retry-aware behavior<\/li>\n<li>The <code>--retry-failed-tests 3<\/code> flag tells the test platform to retry failed tests up to 3 times<\/li>\n<li>The <code>condition: succeededOrFailed()<\/code> ensures test results publish even if tests fail<\/li>\n<\/ul>\n<h3>Understanding the retry outcomes<\/h3>\n<p>Let&#8217;s see how the Azure DevOps UI displays results across different retry scenarios. Imagine a test suite with one stable test that always passes and one flaky test with intermittent failures.<\/p>\n<h4>Scenario 1: All tests pass on first attempt<\/h4>\n<p>When all tests pass immediately, you see standard success indicators. The <code>PublishTestResults<\/code> task completes successfully.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/dotnet\/wp-content\/uploads\/sites\/10\/2025\/12\/passed-on-first-run.webp\" alt=\"Azure DevOps Tests UI showing tests passing on first run\" \/><\/p>\n<h4>Scenario 2: Test passes after retry<\/h4>\n<p>This is where the new behavior shines. The UI shows the test failed on attempts 1 and 2 but succeeded on attempt 3. Even though there were initial failures, the <code>PublishTestResults<\/code> task passes because the test eventually succeeded. This is the correct behavior for handling flaky tests.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/dotnet\/wp-content\/uploads\/sites\/10\/2025\/12\/passed-on-re-run.webp\" alt=\"Azure DevOps Tests UI showing tests passing on re-run\" \/><\/p>\n<h4>Scenario 3: Test fails all retry attempts<\/h4>\n<p>When a test fails across all attempts, the UI groups all failures together for easy review. You can examine each individual failure to identify patterns or differences. The <code>PublishTestResults<\/code> task fails as expected when <code>failTaskOnFailedTests<\/code> is true.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/dotnet\/wp-content\/uploads\/sites\/10\/2025\/12\/failed-on-all-runs.webp\" alt=\"Azure DevOps Tests UI showing tests failing for all attempts\" \/><\/p>\n<p><div class=\"alert alert-success\"><p class=\"alert-divider\"><i class=\"fabric-icon fabric-icon--Lightbulb\"><\/i><strong>Why This Matters<\/strong><\/p>The distinction between VSTest and Microsoft.Testing.Platform is important here: VSTest never had built-in retry support, so this scenario couldn&#8217;t occur naturally. With Microsoft.Testing.Platform&#8217;s Retry extension, you get automatic retry handling with proper Azure DevOps integration; no custom scripting required.<\/div><\/p>\n<h2>Get Started Today<\/h2>\n<p>Azure DevOps now has full support for Microsoft.Testing.Platform\u2014from running tests to publishing retry results with intelligent handling.<\/p>\n<p><strong>Ready to migrate?<\/strong><\/p>\n<ol>\n<li>Update to .NET SDK 10<\/li>\n<li>Add the Retry extension to your test project<\/li>\n<li>Set <code>AllowPtrToDetectTestRunRetryFiles: true<\/code> in your pipeline<\/li>\n<li>Run your tests with <code>--retry-failed-tests<\/code> flag<\/li>\n<\/ol>\n<p><strong>Need help?<\/strong> We&#8217;re here for you:<\/p>\n<ul>\n<li>Ask questions: <a href=\"https:\/\/github.com\/microsoft\/testfx\/discussions\/new\/choose\">GitHub Discussions<\/a><\/li>\n<li>Report issues: <a href=\"https:\/\/github.com\/microsoft\/testfx\/issues\/new\/choose\">GitHub Issues<\/a><\/li>\n<li>Learn more: <a href=\"https:\/\/learn.microsoft.com\/dotnet\/core\/testing\/microsoft-testing-platform-intro\">Microsoft.Testing.Platform documentation<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Azure DevOps enhanced support for Microsoft.Testing.Platform, from running tests to publishing results!<\/p>\n","protected":false},"author":181791,"featured_media":59197,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[685,756,636,7199],"tags":[4,7265,8112,7650,46,73,136,145],"class_list":["post-59196","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dotnet","category-csharp","category-fsharp","category-visual-basic","tag-net","tag-announcements","tag-azdo","tag-azure-devops","tag-c","tag-f","tag-testing","tag-visual-basic"],"acf":[],"blog_post_summary":"<p>Azure DevOps enhanced support for Microsoft.Testing.Platform, from running tests to publishing results!<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/59196","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/users\/181791"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/comments?post=59196"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/59196\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media\/59197"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media?parent=59196"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/categories?post=59196"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/tags?post=59196"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}