What’s New in Our Code Coverage Tooling?

19 comments

Discussion is closed. Login to edit/delete existing comments.

• 

  Heinrich Moser December 22, 2023 6:11 am 13

  collapse this comment copy link to this comment

  Looks great! Now, can we please have Code Coverage support for Visual Studio Professional?

• 

  silkfire December 29, 2023 7:16 am 5

  collapse this comment copy link to this comment

  How does Microsoft.CodeCoverage compare to Coverlet?

  • 

    Chris DaMour December 30, 2023 1:38 pm 1

    collapse this comment copy link to this comment

    came here to ask this same thing. an analysis from MS perspective would be very valuable

  • 

    Jakub Chocholowicz January 2, 2024 1:58 am 2

    collapse this comment copy link to this comment

    If

    1) you don’t need to know how many times each line was executed
    2) it’s not mandatory for you to use only open-source tools in your repository
    3) you don’t need to get coverage in a constrained environment like the case of System.Private.Corelib.dll where there is no support of BCL

    we recommend switching to Microsoft.CodeCoverage for better performance and reliability.

    Please check below links to see performance comparison between Coverlet and Microsoft.CodeCoverage:
    https://github.com/dotnet/roslyn-analyzers/pull/6211#issuecomment-1403679164
    https://github.com/dotnet/roslyn-analyzers/pull/6211#issuecomment-1409894782

• 

  Paulo Morgado January 1, 2024 3:44 am 2

  collapse this comment copy link to this comment

  Will this replace coverlet in the unit test templates?

  • 

    Jakub Chocholowicz January 2, 2024 2:14 am 1

    collapse this comment copy link to this comment

    Microsoft.CodeCoverage is a dependency of Microsoft.NET.Test.Sdk. It means that both tools Microsoft.CodeCoverage and Coverlet are installed by unit test templates. After generating new project you can execute:

    dotnet test --collect "Code Coverage"

    to run tests under Microsoft.CodeCoverage or:

    dotnet test --collect "XPlat Code Coverage"

    to run tests under Coverlet.

    • 

      Paulo Morgado January 5, 2024 1:37 am 1

      collapse this comment copy link to this comment

      Thanks Jakub,

      My question was more if templates dotnet new mstest and the others will be using Microsoft.CodeCoverage instead of Coverlet.

• 

  Michael Dietrich January 3, 2024 1:06 am 0

  collapse this comment copy link to this comment

  This looks really great.
  But something that confuses me a lot is when do I need dotnet-coverage collect and when is dotnet test --collect enough?
  In some samples both are used in others not. Unfortunately it’s not clear to me.

  And since Microsoft.CodeCoverage is a dependency of Microsoft.NET.Test.Sdk, there is no need to explicitly reference Microsoft.CodeCoverage for test assemblies, since it is already referenced transitively?

  • 

    Jakub Chocholowicz January 3, 2024 2:08 am 3

    collapse this comment copy link to this comment

    dotnet-coverage collect collects code coverage for any managed tree of processes (ASP.NET server, console application). dotnet test --collect is specific for test projects and runs tests with code coverage collector enabled. Generally using dotnet test --collect is enough in most scenarios and it is recommended.

    If your test project communicates with external service which is started before test execution, then to get code coverage for external service you have to use dotnet-coverage tool. This scenario you can find here: https://github.com/microsoft/codecoverage/blob/main/samples/Calculator/scenarios/scenario14/README.md

    I understand your confusion here as running tests (dotnet test) is also managed process so you can do: dotnet-coverage collect "dotnet test" and the result is equivalent to dotnet test --collect "Code Coverage". Command dotnet-coverage collect "dotnet test" can be used as workaround for issue: https://github.com/microsoft/vstest/issues/2378 and it is documented here: https://github.com/microsoft/codecoverage/blob/main/samples/Calculator/scenarios/scenario25/README.md

    You are right regarding dependencies. It is enough to have in csproj Microsoft.NET.Test.Sdk. Our recommendation is to keep it up to date as we are constantly improving code coverage.

    • 

      Christian Andritzky January 9, 2024 11:36 pm 0

      collapse this comment copy link to this comment

      What happens when you run

      dotnet test --collect "Code Coverage"

      and the unit test project starts (and gracefully terminates) the server (as a child process)? Will Microsoft.CodeCoverage collect the code coverage of the server (child) process? coverlet does this if (and only if) the server (child) process is properly terminated.

      • 

        Jakub Chocholowicz January 9, 2024 11:53 pm 1

        collapse this comment copy link to this comment

        Microsoft.CodeCoverage and dotnet-coverage tools collect code coverage for all child processes. Additionally coverage data is passed through shared memory and coverage is collected even if child process crashes. We don’t store any information on termination. Example is here: https://github.com/microsoft/codecoverage/blob/main/samples/Calculator/scenarios/scenario11/README.md

• 

  Michael Dietrich January 5, 2024 2:35 am 2

  collapse this comment copy link to this comment

  I realized the resulting code coverage reports also contain code coverage information for external code such as MassTransit.
  Is there an easy way to only collect code coverage information for code files that actually exist locally? Since depending on the referenced NuGets, this can fairly distort the resulting code coverage report. E.g. in the mentioned case I can see several entries like /_/src/MassTransit.Abstractions/Attributes/ConfigureConsumeTopologyAttribute.cs which we would like to filter out.

  • 

    Jakub Chocholowicz January 8, 2024 12:20 am 1

    collapse this comment copy link to this comment

    If pdb is found, we add code coverage for module into report. Currently we don’t have any logic to exclude based on existence of sources (like coverlet). We plan to fix it under https://github.com/microsoft/codecoverage/issues/72

    As workaround you can manually exclude third party libraries or configure regex to match only your dlls. Please check this example: https://github.com/microsoft/codecoverage/tree/main/samples/Calculator/scenarios/scenario06

    • 

      Gioacchino Piazzolla January 8, 2024 5:59 am 0

      collapse this comment copy link to this comment

      This would be a huge improvement 🙂

    • 

      Jakub Chocholowicz January 24, 2024 3:56 am 0

      collapse this comment copy link to this comment

      Logic to exclude based on existence of sources (like coverlet) was added. It will be available in 17.10.0 package. Check: https://github.com/microsoft/codecoverage/issues/12#issuecomment-1907966480

• 

  Christian Andritzky January 9, 2024 11:43 pm 0

  collapse this comment copy link to this comment

  “Auto-Merge for solutions” is a great and very welcome addition. But will it work when there are test projects with multi-targeting running tests e.g. for .NET Framework and modern .NET? We use this a lot while we are migrating a huge application from .NET Framework to .NET 6 (or 8). AFAIK you cannot execute .NET Framework tests and .NET (Core) tests within the same test host process. You need two separate runs/processes for this.

  How does this work with the new CodeCoverage features? Is there a sample for this scenario?

  • 

    Jakub Chocholowicz January 9, 2024 11:50 pm 1

    collapse this comment copy link to this comment

    Yes, this is supported. You are right that each test project is running on specific test host process. After all projects are done code coverage reports for all projects are merged together. We don’t have sample for this.

• 

  Jeff Chen February 6, 2024 5:02 am 0

  collapse this comment copy link to this comment

  This is a great update. I am trying to use this package in my azure pipeline. The question is that how to spefiic ‘Format’ in dotnet test task via a command line ?

  When I try this in ‘Arugments’ field of dotnet test task:

  --no-build --collect "Code Coverage;Format=cobertura;"

  the task will show following message:

  Data collection : Unable to find a datacollector with friendly name 'Format=cobertura'.

  However, in the document here:
  https://github.com/microsoft/codecoverage/blob/main/README.md
  It shows a sample code like this:

  dotnet test --collect "Code Coverage;Format=cobertura"

  So, it looks like the approach used in command line cannot be used in dotnet test task ?

  What is the correct way to use ‘Format’ parameter in this task? thanks a lot

  • 

    Jakub Chocholowicz February 12, 2024 12:36 am 0

    collapse this comment copy link to this comment

    What version of .NET SDK you are using? I think updating it will help.