Code Coverage now available for PowerShell Core!

James.W.Truher

This is the first of a series of posts on PowerShell Core and the tools we use to test it. If you’ve looked at the main project for PowerShell (https://github.com/PowerShell/PowerShell, you may have noticed a new badge down in the Build status of nightly builds:

We are supplying code coverage numbers for our test pass via the OpenCover project (https://github.com/OpenCover/opencover) and we visualize our code coverage percentage via coveralls.io (https://coveralls.io/github/PowerShell/PowerShell?branch=master). This means you can see some details about our testing and how much of PowerShell is covered by our test code.

You can get your own coverage numbers easily via our OpenCover module which may be found in the <RepoRoot>test/tools/OpenCover directory. To generate a code coverage report, you need to create a build which supports code coverage. Currently, that’s only available on Windows, but we do have an easy way to get it:

(All of these commands assume that you are at the root of the PowerShell repo)

# create a code coverage build.
PS> Start-PSBuild -Configuration CodeCoverage -Publish

# Now that you have a build, save away the build location
PS> $psdir = split-path -parent (get-psoutput)

# Import the OpenCover Module
PS> Import-module $pwd/test/tools/OpenCover

# install the opencover package
PS> Install-OpenCover $env:TEMP

# now invoke a coverage test run
PS> Invoke-OpenCover -OutputLog Coverage.xml -test $PWD/test/powershell -OpenCoverPath $env:Temp/OpenCover -PowerShellExeDirectory $psdir

If you want to get code coverage for only the tests that we run in our Continuous Integration (CI) environment, add the parameter -CIOnly. Then you’ll need to wait for a bit (on my system and using -CIOnly, it takes about 2.5 hours to run).

Looking at the Data

The OpenCover module can also help you visualize the results from a very high level.

# first collect the coverage data with the Get-CodeCoverage cmdlet
PS> $coverData = Get-CodeCoverage .\Coverage.xml
# here’s the coverage summary
PS> $coverData.CoverageSummary
NumSequencePoints       : 309755
VisitedSequencePoints   : 123779
NumBranchPoints         : 105816
VisitedBranchPoints     : 39842
SequenceCoverage        : 39.96
BranchCoverage          : 37.65
MaxCyclomaticComplexity : 398
MinCyclomaticComplexity : 1
VisitedClasses          : 2005
NumClasses              : 3309
VisitedMethods          : 14912
NumMethods              : 33910

# you can look at coverage data based on the assembly
PS> $coverData.Assembly | ft AssemblyName, Branch, Sequence

AssemblyName                                     Branch Sequence
------------                                     ------ --------
powershell                                       100    100
Microsoft.PowerShell.CoreCLR.AssemblyLoadContext 45.12  94.75
Microsoft.PowerShell.ConsoleHost                 22.78  23.21
System.Management.Automation                     41.18  42.96
Microsoft.PowerShell.CoreCLR.Eventing            23.33  28.57
Microsoft.PowerShell.Security                    12.62  14.43
Microsoft.PowerShell.Commands.Management         14.69  16.76
Microsoft.PowerShell.Commands.Utility            52.72  54.40
Microsoft.WSMan.Management                       0.36   0.65
Microsoft.WSMan.Runtime                          100    100
Microsoft.PowerShell.Commands.Diagnostics        42.99  46.62
Microsoft.PowerShell.LocalAccounts               0      0
Microsoft.PowerShell.PSReadLine                  6.98   9.86

I’m not going to go through all the different properties that are reported, we’ll take a closer look at those in future posts.  The Get-CoverageData cmdlet is still fairly rudimentary, but it will provide some details. This is part of our public repo, so I encourage you to enhance it and log issues if you find them!

Better Coverage Visualization

Another way to view coverage data is via the ReportGenerator package, which creates HTML reports and provides much more details about the coverage. The ReportGenerator package is available via the find-package cmdlet in the PackageManagement module. The following will install the package, and show how to run it:

# find and install the report generator package
PS> find-package ReportGenerator -ProviderName nuget -Source https://nuget.org/api/v2 | install-package -Scope CurrentUser
PS> $ReportGenExe = “$HOME\AppData\Local\PackageManagement\NuGet\Packages\ReportGenerator.2.5.2\tools\ReportGenerator.exe”
# invoke the report generator and create the report in c:\temp\Coverage
PS> & $ReportGenExe -reports:Coverage.xml -targetdir:c:\temp\Coverage

Now that you’ve created the reports, you can visualize them with your browser.

PS> invoke-item C:\temp\Coverage\index.htm

Click on the “Enable filtering button”, and then “Collapse all” and you should see something similar to:

You can then drill in on what interests you (Microsoft.PowerShell.Commands.Utility, for example)

Of course, there’s a lot more detail to discover, and I encourage you to poke around. In my next post, I’ll go through an entire workflow:

• Select an area for improvement
• Create new tests
• Gather new coverage data
• Compare results from previous runs

I’ll target something specific (a cmdlet) and show how to determine the gaps and how to fill them.

Call To Action!

Now that you see how easily you can generate code coverage data, this is a great opportunity to provide some additional coverage and increase the quality of our releases. If you see some area which you’re passionate about or notice an area which you would like to measure better, it’s a great way to provide improved coverage. As you create new PRs, you can aim for high coverage in your new functionality (85-90%), and now you can measure it!

James.W.Truher

Follow