Part 3: Unit testing with Traits and code coverage in Visual Studio 2012 using the TFS Build – and the new NuGet adapter approach

The Unit Testing series by one of the ALM MVPs Terje Sandstrom has been some of the highest read posts on the ALM blog…In this post he shows how to easily provide unit Test Adapters from NuGet.

Thanks for the Great Post Terje!

 

**********************************************************

This is the 3rd post in a series on Unit Testing in Visual Studio 2012, this one focusing on how to use the new features with the TFS Build system, but also some news on unit testing in Visual Studio itself, in particular the use of NuGet as an engine for providing the adapters.  We will also cover the setup and customization of code coverage in TFS.

The former posts are:

How to manage unit tests in Visual Studio 2012 Update 1 : Part 1–Using Traits in the Unit Test Explorer

Part 2–Using Traits with different test frameworks in the Unit Test Explorer

The major change from 2010 is the multi-framework testing capabilities, allowing other test frameworks to be used along with the built-in ones, integrating them and their results with a very good blend.  This capability is also valid for the TFS Build system.  There are certain differences which will be outlined in this post.

 

How traits work on TFS

The testing system for TFS is not identical to the one in Visual Studio.  The trait support is different, and requires you to set up things a bit differently.

There is really no traits in TFS Build, but it matches up with a set of TFS Build Properties. These properties can be used for Test Case Filtering.  We will walk through how you match up these properties with the possible set of traits that the adapters can provide.

 

Test case filtering

TFS Build has a system for test case filtering, that can use properties from the tests.  These properties can match both traits or some of the other test properties from a test.

The test case filtering system is described in this blog post. The filter syntax here is different from the one used in the Test Explorer, but has a very easy to learn syntax. The major point is that the properties used in the filter must ALL be implemented by the test adapter in use for any given test run.  In a  given test run you can not mix properties from different adapters, they must be implemented in all.   The NUnit adapter and the upcoming XUnit adapter has been made to match the properties from MSTest. This means that these adapters can be used together in the same run with no conflicts.

The limitation here is that you can not use any other properties than those given by the MSTest adapter and supported by the TFS Build, shown below (cut from blog post above).  The Name and FullyQualifiedName are the only ones that are consistently equal for all adapters.

There is a conversion built into the NUnit adapter and the upcoming XUnit adapter, which makes the TestCategory and Priority match the conventions used in these frameworks.  For an upcoming Chutzpah adapter the Module keyword is made to match the TestCategory. The matches is shown in the “Traits summarized” section in Part 2, shown in the table below, but converted to match the conventions in TFS Build.

 

TFS Build property name	MSTest	Native C++	NUnit	XUnit	Chutzpah
Priority	Priority	TEST_PRIORITY	Priority	Trait(“Priority”,value)	—–
TestCategory	TestCategory	—–	Category	Trait(“Category”,value)	Module

As an example, this means you can keep your NUnit Category attribute in the source code, and filter this on TFS using the TestCategory property.  The adapter will make these match up.

Be aware that you can specify a category for Native C++ as TEST_METHOD_ATTRIBUTE (“Category”,value), which will work for Visual Studio (see part 2) but not for TFS Build. The built-in adapter for Native C++ does not match up this with the TFS Category property.

Further:  If a wrong property is specified, the test run will give an error for the assemblies and adapter that fails, but the other assemblies will still run. If we have a filter set up with f.e.  TestCategory=CI, and then add a native CPP project to the solution, we will end up with the following error message from the TFS Build: 

This is because the Native CPP adapter doesn’t support the TestCategory, and means means these can’t be used in the same test run.  So…  in order to handle this, we must define multiple test runs. 

Using multiple test runs

You can specify your tests from different test frameworks to run in different test runs.  Each test run can then use a set of properties that match the tests for the adapter you want to run in that particular test run.  The caveat here is that you need to implement a naming convention for the tests so that you can filter out the test assemblies based on that, using the   If not, you will get the error message above for every run that doesn’t support one of your properties.

To set up a secondary test run, one add it as shown in the figure below:

Select the Automatest Tests line, the small […] button then appears to the far right, pressing that takes you to the dialog for managing test runs.  Press the Add button to add another test run.

To filter the assemblies, one uses the Test Sources Spec property for the Test Source, in this dialog it is called “Test assembly file specification”.

In the case with the Native CPP, the important thing is to make sure that those tests are not included with the standard run. That means the Native CPP must have a suffix of the name that doesn’t match the primary test source spec.

So, we rename the CPP Native test to end with “cppunittst”.  This will not match the original filter of *test*.dll.  This Test Source Spec filter acts before the Test Case Filter, and controls which assemblies are sent down to the adapter. The Test Case Filter works on the adapter afterwards to further limit the set of tests.  Notice that in this case we have set up a testcasefilter to match up with any tests which FullyQualifiedName (FQN) contains the string “CPP”.  The FQN matches the

The results for the runs, and details for the 2nd run:

There is currently a know bug for the ID field for CPP projects.  All methods are prefixed with “DefaultClassName” instead of the correct one, as shown below for managed code tests.  

The FQN filter however uses the correct Fully Classified Name and now that is shown in the ID field, so the functionality works as it should.

Enabling Code Coverage in TFS Builds

Enabling code coverage in TFS Builds are very easy.  The code overage can be enabled from the Test Run definition dialog, using either the default settings

 

Running a build with this enabled, will result in a new section in the summary build results,

and clicking the Coverage Results will first ask you to download the file, answer Open it, and it then open in Visual Studio. 

Note that the NUnit and XUnit tests are also automatically included in the same output, this is due to the new Test Runner.

Note:  If you have both Visual Studio 2010 and 2012 installed, it may open it in VS 2010.  Check and correct the .coverage extension to what you want under Default Programs in the Control Panel

 

 

If you need to customize the code coverage setup, choose Custom, and add a .runsettings file.

To customize the code coverage you must add a .runsettings file. 

You can either install an Item Template for the runsettings file from the Visual Studio Gallery to add this ,

or do it manually by going to Customizing Code Coverage Analysis and copy out the sample .runsettings xml from the bottom there.  Always start out with this file, it contains a lot of stuff that comes in very handy.  Modify the file to suit your own preferances.

In the .runsettings file you can add exclusions for assemblies you don’t want included.  A typical one is to exclude the test assemblies itself, as they are not part of the final release code.  You then reuse the special suffixes you have created for these and add them into the Exclude  clause using a regular expression under ModulePaths.

<ModulePaths>
     <Exclude>
            <!-- Exclude all file paths that end in "Test" or "CPPUnitTst": -->
            <ModulePath>.*Test.dll</ModulePath> 
             <ModulePath>.*CPPUnitTst.dll</ModulePath> 
      </Exclude>
</ModulePaths>

Be aware that the ModulePath is absolute paths, so be careful with patterns like “.*Test.*”.  

If you get issues with the code coverage settings, see this troubleshooting guide.

 

Providing 3rd party adapters through NuGet

The 3rd party adapters has earlier been provided as VSIX packages only, available through the Visual Studio Gallery.  That has been fair and worked well, but has been awkward for using on a TFS Build server, since there is no host there for the VSIX.  The approach then has been to extract the content of VSIX and adding that to a custom build activities folder in source control. The Build Controller can then get the adapters from this location.  This procedure has been described in the ALM Rangers Build Customization Guide, page 33, downloadable from their codeplex site, or go to this page to see the procedure.

From update 1 there has been a little known possibility to use NuGet to provide the adapters, but no 3rd party adapters have used this approach until now.  The NUnit Adapter is the first one to be available as a NuGet adapter, and we will use that as an example in this post.  We have been informed that XUnit will also arrive very soon on NuGet with their adapter.

Using NuGet as the adapter provider has several advantages, the most important one is:

• No need to install anything on either TFS Build Server or on any developers machine.

Both Visual Studio 2012 (Update 1 or later) and TFS Build is NuGet test adapter aware !

Since NuGet installs the adapters into the solution itself, the adapters follow the solution, and the tests will “just work” without any need from the developer to install anything. 

NUnit Adapter on NuGet

The NUnit adapter, from version 0.95,  comes in two flavors:

1. The NUnit Test Adapter package which contains only the adapter
2. The NUnit Test Adapter with NUnit Framework version 2.6.2

If you choose to use flavor 1, you will need to also install the NUnit framework package.  Note that you only need the adapter in one of your projects, but it doesn’t matter if it is in multiple.   Use this approach if you already have the NUnit framework installed,  otherwise choose flavor 2.  You will see a reference in each project to the NUnit Testadapter, but the adapter itself will not be copied to your build output (it has its copy local set to false).  If it bothers you to see this in all the projects you can use flavor 1 on one project only.  That is enough to make it work for all the projects.

Note that the adapters will continue to be available through VS Gallery if you prefer to have it installed on your machine that way.  And, there is no crash between the VSIX and the NuGet adapters, as a developer you don’t need to choose between them. 

One thing to be aware of is that the VSIX adapter takes precedence over the NuGet adapter, of you have both installed.  Even if you update the NuGet adapter to a later version than the VSIX, the VSIX will be used.   That means you might run on an earlier version of the adapter, whereas TFS Build and developers who don’t have the VSIX will use the newer NuGet version. 

On the other hand, given latest versions, having the adapters in the solution make the solution more robust, as any developer with or without having the correct adapter can use the tests with no further action to be taken. 

Also be aware that what you place in the custom activities folder for TFS Build will take precedence over whatever is included as a NuGet package.  In this case it means that you may have upgraded and running on a later version, whereas the TFS Build will run on an earlier version.

The adapter placed in the custom activities folder will apply for the given Build controller and all agents running under that build controller.  So if you have one build controller for your Team Project Collection, that adapter will apply for all team projects within that collection.

In order to keep this under control, you should verify the version the adapter is running under.

The NUnit adapter will inform you in the Output/Tests window which version it is running.

In a TFS build, you open the build results, shown below,  click the test results summary to get the test results details, and click the Test run status to get to the Result Summary with the Standard Console Output, where the version number can be seen, together with the assemblies that have been checked.

 

Note that the Package restore will only get you the exact version you installed the first time.  You must explicitly update the NuGet packages too, using either the Package Manager Console with explicit commands:

or check if there is anything in the Updates tab in the Manage NuGet Packages for Solution dialog.

 

Other adapters on NuGet

Of the other adapters mentioned in earlier posts, the state currently is:

XUnit will be provided through NuGet real soon too. 

Chutzpah has a design which doesn’t make it possible to be deployed through NuGet.  This means you need to follow the guidelines for adding it to your custom activities folder in source control to make it work on TFS Build. 

MbUnit, unknown at the time of writing.

 

Links to other MSDN articles on Tests in TFS Build

Run Tests in Your Build Process

Build with other unit test frameworks

Customizing Code Coverage Analysis

Using Code Coverage to Determine How Much Code is being Tested

How to: Deploy Files for Tests

2: Unit Testing: Testing the Inside

 

Written By

Terje Sandstrom

—— Chief Software Geek at Inmeta Consulting in Scandinavia —– and a Visual Studio ALM MVP