{"id":3903,"date":"2012-11-20T12:24:02","date_gmt":"2012-11-20T12:24:02","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/2012\/11\/20\/part-2using-traits-with-different-test-frameworks-in-the-unit-test-explorer\/"},"modified":"2024-03-13T08:57:39","modified_gmt":"2024-03-13T16:57:39","slug":"part-2using-traits-with-different-test-frameworks-in-the-unit-test-explorer","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/devops\/part-2using-traits-with-different-test-frameworks-in-the-unit-test-explorer\/","title":{"rendered":"Part 2\u2013Using Traits with different test frameworks in the Unit Test Explorer"},"content":{"rendered":"

As promised one of the ALM MVPs, Terje Sandstrom, has delivered Part 2 of his Unit Test Series: \u201cUsing Traits with different test frameworks in the Unit Test Explorer\u201d.<\/p>\n

From the traffic and comments on his first post these are clearly very appreciated articles!<\/p>\n

Thanks Terje!<\/p>\n

\u00a0<\/p>\n

*********************************************************************************************************************************************\u2019<\/p>\n

In Part 1<\/u><\/a>\u00a0 traits in general was described, and we used examples from using MSTest as a testing framework.\u00a0 In this part we will built upon what we did in Part 1, but extend it with many of the other frameworks.\u00a0 The Unit Test Explorer is test framework agnostic!\u00a0 It works with any framework, as long as there exist an adapter, and such adapters are coming for NUnit, XUnit, MBUnit, Javascript-Chutzpah, MS Native C++ and even more.\u00a0 A list of these and links to information and downloads for this can be found in <\/u>this blog post<\/u><\/a>. Note that some of these are being updated along with the release of Update 1. See summary at the end of the blog post.<\/p>\n

The summary at the end also shows the syntax for the different traits in the different test frameworks.<\/p>\n

We will continue with the code examples from Part 1, and do something that perhaps is not so usual, mixing in some of the other frameworks and let them play together !\u00a0 That way it is also easier to see the differences between these.\u00a0 It will be done step by step, so the examples expand as we add more.\u00a0 The sequence is completely random and doesn\u2019t express any preferences.<\/p>\n

What is pretty cool, is that we show that in principle all types of tests can be run at the same time,<\/strong> many even in the same project, although Managed C++, Native C++ and Javascript tests stay in their own projects, but all in the same solution, and the results will be merged together in the Test Explorer !<\/p>\n

The different frameworks and their adapters are at somewhat different levels regarding how compliant they are to the new Trait system.\u00a0 Information is given for each framework on what is missing, and also expected time of arrival for the improvements where we have managed to obtain that information.<\/p>\n

\u00a0<\/p>\n

Adding in NUnit<\/u><\/a><\/font><\/strong><\/p>\n

Now let us mix in a NUnit Test first. In order to do this you must install the NUnit Test Adapter from Visual Studio Gallery<\/u><\/a>, and you must add a NUnit package from NuGet to your project. Adding the NuGet is done in a few steps as shown in the image series below.<\/p>\n

Step A: From Visual Studio Tools menu, choose Library Package Manager\/Manage NuGet Packages for Solution.<\/p>\n

\"image_thumb4\"<\/a><\/p>\n

Step B: If this is the first time you add this to any solution, choose Online, and add \u201cNUnit\u201d in the search box, select the NUnit. (Note, you don\u2019t need the NUnit.Runners, as VIsual Studio has its own runner that will run these for you). If you already have done this once, go instead to the Installed Packages, NUnit will be listed there and press the button \u201cManage\u201d. Any of these will take you to step \u2018C\u2019.<\/p>\n

\"image_thumb5\"<\/a><\/p>\n

Step C: Choose the test projects where you want to include NUnit.<\/p>\n

\"image_thumb6\"<\/a><\/p>\n

Your project will then look like this, where (1) is included Dll and the NuGet package configuration file. (If you ever want to remove it, these two files are the ones to remove. If you got more packages, you must open and edit the packages.config.)<\/p>\n

\"SNAGHTML1b4cca11_thumb1_thumb\"<\/a><\/p>\n

Now you can add a new NUnit test file to your project, 2) in the screenshot above.<\/p>\n

NUnit has a different way of setting up categories, in fact it can be done in two different ways. You can add the category as a text string, the same way as you do it in MSTest above, or you can derive a new class from the Unit CategoryAttribute class. We have done both things below:<\/p>\n

\"image_thumb7_thumb\"<\/a><\/p>\n

When opening up the Test Explorer now we see the following:<\/p>\n

\"image_thumb9_thumb\"<\/a><\/p>\n

Both methods are in the same Trait, named Category [CI], even if they were defined differently. That is pretty cool !<\/p>\n

Notice that the MSTest test that is in the CI category is placed in its own group.\u00a0 The categories from NUnit and MSTest are different even if they are spelled equally.\u00a0 That, however, only applies for categories.<\/p>\n

If we add in another test method and decorate that with the same kind of attributes that we did in the MSTest example in Part 1:<\/p>\n

\"image_thumb8\"<\/a><\/p>\n

In NUnit there are no explicit Owner and Priority property, but you can use the general Property attribute instead.\u00a0 The NUnit Property is more advanced than the MSTest Property, it has overloads for string, int and double, so implementing Priority with this makes it equal to the MSTest specific Priority.<\/p>\n

The output from this then becomes:<\/p>\n

\"image_thumb11\"<\/a><\/p>\n

Note that except for the categories, all the other properties are collected together for both MSTest and NUnit.<\/p>\n

Status<\/strong><\/p>\n

    \n
  1. Current version of the adapter at time of writing is 0.93.\u00a0 There is a bug concerning interoperability with Native C++<\/u><\/a> which is fixed in the upcoming 0.94 version, ETA Nov 30th 2012. <\/li>\n
  2. Version 0.94 will also add class based Categories, see discussion below under xUnit.\u00a0 <\/li>\n<\/ol>\n

    \u00a0<\/p>\n

    Adding in xUnit.net<\/u><\/a><\/font><\/strong><\/p>\n

    The xUnit.Net adapter<\/u><\/a> can also be found at Visual Studio Gallery (traits implemented in version 0.9.3), and the framework is found on NuGet the same way, search for \u201cxUnit\u201d, and install both the same way as done for NUnit, adding the libraries to the test project.\u00a0<\/p>\n

    \"image_thumb91\"<\/a><\/p>\n

    In the xUnit.net framework, the different traits are implemented by decorating the test method with the Trait attribute.\u00a0 This is used for all kind of properties.\u00a0 Everything is a Trait in xUnit.\u00a0 It doesn\u2019t have any int overloads, so the priority must use a string representation, but the Test Explorer will treat that string and the MSTest int as the same.<\/p>\n

    The same code then becomes:<\/p>\n

    \"image_thumb3\"<\/a><\/p>\n

    You can also derive new classes from the TraitAttribute class, so if you like you can implement for example the Category, specific categories you want, and the Priority attributes that way:<\/p>\n

    \"image_thumb12\"<\/a><\/p>\n

    The results with both NUnit, MSTest and XUnit running at the same time is then:<\/p>\n

    \"image_thumb18\"<\/a><\/p>\n

    Both the NUnit and XUnit categores fall sweetly into the Categories group, MSTest are in its own special category types (CI, Developer and Production).\u00a0 They all treat the other traits equally. Notice that the special new attributes made above for \u2018CI\u2019 and \u2018Priority\u2019 are treated the same as the original ones.<\/p>\n

    Class based attributes<\/strong><\/p>\n

    When I write unit tests I often group them by class.\u00a0 I rarely put two different types of unit tests in the same class.\u00a0 So, what I would like is to be able to decorate the class with Traits too.\u00a0 Those traits should then be treated as applying to all the methods in that class.\u00a0 It turns out that XUnit is the only framework I have tested so far that have this capability.\u00a0 MSTest refuses you to add the Traits to the class, they are bound to methods only.\u00a0 NUnit allows you to do it, but these traits have no effect yet, but the next version of the NUnit adapter should have this capability too for categories.<\/p>\n

    We add some new test classes to all the projects, looking like this:<\/p>\n

    \"image_thumb20\"<\/a><\/p>\n

    (Equally for NUnit and MSTest, except different suffices, N11\/N12 and Ms11\/Ms12.<\/p>\n

    And the results show that the NUnit and MSTest ends up the No Traits group, whereas the XUnit comes in correctly<\/p>\n

    \"image_thumb23\"<\/a><\/p>\n

    \u00a0<\/p>\n

    Adding in MbUnit<\/u><\/a><\/strong><\/font><\/p>\n

    To add in MbUnit one need to add first the Gallio framework<\/u><\/a>, then the MbUnit adapter<\/u><\/a>, and finally using NuGet to get the Gallio & MbUnit dll\u2019s into your project:<\/p>\n

    \"image_thumb26\"<\/a><\/p>\n

    \u00a0<\/p>\n

    MbUnit has specific Traits, which matches some of the others, but not all.\u00a0 It has Category, Author and Importance.<\/p>\n

    \"image_thumb41\"<\/a><\/p>\n

    MBUnit also allows for derivation of new attribute classes like NUnit and xUnit.net, but the adapter doesn\u2019t support this for traits.<\/p>\n

    One important thing to notice is that the adapter don\u2019t accept the Importance attribute shown above.\u00a0<\/p>\n

    The current version of Gallio\/MbUnit, version 3.4 does not work correctly with its own version 3.4 of VS Test Adapter, giving an exception when you try to run it. The adapter seems to have been built for an earlier version.<\/p>\n

    I did download their adapter code and made a few corrections and added the Trait code, just to see how the underlying MbUnit\/Gallio treats this.\u00a0 The adapter is only a bridge between the test runner and the test framework.\u00a0 I will try to get these code changes back up to the MbUnit project, so it should be ready to go in the next release.<\/p>\n

    \"image_thumb39\"<\/a><\/p>\n

    Notice that we now are running 3 adapters at the same time – cool !<\/p>\n

    Status<\/strong><\/p>\n

    Current version of the adapter (3.3) does not support Traits, but next version of the adapter will do so.<\/p>\n

    \u00a0<\/p>\n

    Adding in C++<\/strong><\/font><\/p>\n

    Managed C++<\/font><\/strong><\/p>\n

    Test for Managed C++ is built-in in Visual Studio.\u00a0 It uses the same Traits as for C#. No surprises here.\u00a0<\/p>\n

    \"image_thumb30\"<\/a><\/p>\n

    \u00a0<\/p>\n

    Native C++<\/font><\/strong><\/p>\n

    Test for Native C++<\/u><\/a> is also built-in to Visual Studio 2012.\u00a0 The set of traits available are different from managed, and the way they are set is different from managed C++. You can extend the traits by using the same property\/value mechanism, the attribute to use then is the TEST_METHOD_ATTRIBUTE which takes a property and a value as parameters.<\/p>\n

    \"image_thumb1\"<\/a><\/p>\n

    The attributes that matches the traits are the three shown in the code example above.<\/p>\n

    The result when running this is then :<\/p>\n

    \"image_thumb7[1]\"<\/a><\/p>\n

    \u00a0<\/p>\n

    C++ also has full support for class based and even module based traits.\u00a0 The example code below show a class based trait attribute<\/p>\n

    \"image_thumb10\"<\/a><\/p>\n

    and the result is then that this applies to all methods in this class.<\/p>\n

    \"image_thumb11[1]\"<\/a><\/p>\n

    So the C++ Unit Test framework supports all the traits and on all levels.\u00a0 That\u2019s very cool !<\/p>\n

    \u00a0<\/p>\n

    \u00a0<\/p>\n

    Adding in JavaScript testing with Chutzpah<\/u><\/a><\/strong><\/font><\/p>\n

    Chutzpah is the test adapter<\/u><\/a> for the QUnit<\/u><\/a> and Jasmine<\/u><\/a> test frameworks for JavaScript.<\/p>\n

    QUnit and Jasmine have the keywords \u2019Module\u2019 and \u2018Group\u2019 respectively that can be used to decorate the test methods.\u00a0 Those are the only ones available who can be used to implement Traits.\u00a0 They are semantically not meant like Traits, but can be used that way.<\/p>\n

    Also note that Chutzpah currently runs tests per test file, not per test.\u00a0 So if you select a single test and runs that, all the other tests in the same file are also being executed. This imply you should consider how your Modules and tests are organized by file.\u00a0 One file should not contain more than one Module, if you intend to run them separately.\u00a0<\/p>\n

    Again, I added a few lines trait coding to the Chutzpah source code, and this is how it looks, using Module as the keyword for a Trait.<\/p>\n

    \"image_thumb43\"<\/a><\/p>\n

    And the results, together with the other tests.\u00a0 In the screenshot below, I have chosen to execute the first test in group a, and as can be seen the other tests are also run.<\/p>\n

    \"clip_image001_thumb1\"<\/a><\/p>\n

    \u00a0<\/p>\n

    Status<\/strong><\/p>\n

      \n
    1. The current version at time of writing (Nov. 12th 2012) of Chutzpah is 2.2.1. This version does not support Traits.\u00a0 The next version will support traits. <\/li>\n<\/ol>\n

      \u00a0<\/p>\n

      Summary<\/strong><\/font><\/p>\n

      \u00a0<\/p>\n

      Release schedules for trait aware 3rd party framework adapters<\/strong><\/p>\n\n\n\n\n\n\n\n
      \n Framework<\/strong>\n <\/td>\n\n ETA<\/strong>\n <\/td>\n\n Comment<\/strong>\n <\/td>\n<\/tr>\n
      \n NUnit\n <\/td>\n\n Nov. 30th 2012, version 0.94\n <\/td>\n\n \u00a0\n <\/td>\n<\/tr>\n
      \n xUnit.Net\n <\/td>\n\n Current version 0.9.3\n <\/td>\n\n \u00a0\n <\/td>\n<\/tr>\n
      \n MbUnit\n <\/td>\n\n Next version, date TBD\n <\/td>\n\n Code ready\n <\/td>\n<\/tr>\n
      \n Chutzpah\n <\/td>\n\n Next version, date TBD\n <\/td>\n\n Solution ready, needs coding and some time\n <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

      \u00a0<\/p>\n

      Traits summarized<\/strong><\/p>\n

      The first column merely sets a generic name for the traits, to be able to compare them between the adapters\/frameworks. The \u2018—-\u2018 means \u201cNot implemented\u201d.<\/p>\n\n\n\n\n\n\n\n\n\n\n
      \n Trait \u2018generic\u2019 name<\/strong>\n <\/td>\n\n MSTest (C# and Man C++)<\/strong>\n <\/td>\n\n Native C++<\/strong>\n <\/td>\n\n NUnit<\/strong>\n <\/td>\n\n xUnit.Net<\/strong>\n <\/td>\n\n MbUnit<\/strong>\n <\/td>\n\n Chutzpah<\/strong>\n <\/td>\n<\/tr>\n
      \n Category\n <\/td>\n\n Category\n <\/td>\n\n TEST_METHOD_ATTRIBUTE(\u201cCategory\u201d,) <\/value>\n <\/td>\n\n Category\n <\/td>\n\n Trait(\u201cCategory\u201d,) <\/value>\n <\/td>\n\n Category\n <\/td>\n\n Module\n <\/td>\n<\/tr>\n
      \n Priority\n <\/td>\n\n Priority\n <\/td>\n\n TEST_PRIORITY\n <\/td>\n\n Property(\u201cPriority\u201d,) <\/value>\n <\/td>\n\n Trait(\u201cPriority\u201d,) <\/value>\n <\/td>\n\n —- (Importance<\/strike>)\n <\/td>\n\n —-\n <\/td>\n<\/tr>\n
      \n Owner\n <\/td>\n\n Owner\n <\/td>\n\n TEST_OWNER\n <\/td>\n\n Property(\u201cOwner\u201d,) <\/value>\n <\/td>\n\n Trait(\u201cCategory\u201d,) <\/value>\n <\/td>\n\n Author(kind of)\n <\/td>\n\n —-\n <\/td>\n<\/tr>\n
      \n Property\n <\/td>\n\n Property\n <\/td>\n\n TEST_METHOD_ATTRIBUTE\n <\/td>\n\n Property\n <\/td>\n\n Trait\n <\/td>\n\n —-\n <\/td>\n\n —-\n <\/td>\n<\/tr>\n
      \n \u00a0\n <\/td>\n\n \u00a0\n <\/td>\n\n \u00a0\n <\/td>\n\n \u00a0\n <\/td>\n\n \u00a0\n <\/td>\n\n \u00a0\n <\/td>\n\n \u00a0\n <\/td>\n<\/tr>\n
      \n Support inheritance\n <\/td>\n\n No\n <\/td>\n\n No\n <\/td>\n\n Yes\n <\/td>\n\n Yes\n <\/td>\n\n No\n <\/td>\n\n NA\n <\/td>\n<\/tr>\n
      \n Class based traits\n <\/td>\n\n No\n <\/td>\n\n Yes\n <\/td>\n\n Yes\n <\/td>\n\n Yes\n <\/td>\n\n No\n <\/td>\n\n NA\n <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

      Property is a generic property\/value set, and support for that is very nice to have. Implementing that means the traits are very extensible, and you can twist them to anything you like, and have full support of that in the Test Explorer.<\/p>\n

      Have fun !<\/p>\n

      \u00a0<\/p>\n

      \n
      \n<\/blockquote>\n

      ******************************<\/p>\n

      \u00a0<\/p>\n

      By<\/p>\n

      Terje Sandstrom<\/font><\/a><\/h6>\n

      —— Chief Software Geek at Inmeta Consulting in Scandinavia —– and a Visual Studio ALM MVP<\/p>\n

      <\/p>\n","protected":false},"excerpt":{"rendered":"

      As promised one of the ALM MVPs, Terje Sandstrom, has delivered Part 2 of his Unit Test Series: \u201cUsing Traits with different test frameworks in the Unit Test Explorer\u201d. From the traffic and comments on his first post these are clearly very appreciated articles! Thanks Terje! \u00a0 *********************************************************************************************************************************************\u2019 In Part 1\u00a0 traits in general was […]<\/p>\n","protected":false},"author":77,"featured_media":45953,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-3903","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"acf":[],"blog_post_summary":"

      As promised one of the ALM MVPs, Terje Sandstrom, has delivered Part 2 of his Unit Test Series: \u201cUsing Traits with different test frameworks in the Unit Test Explorer\u201d. From the traffic and comments on his first post these are clearly very appreciated articles! Thanks Terje! \u00a0 *********************************************************************************************************************************************\u2019 In Part 1\u00a0 traits in general was […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/3903","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/users\/77"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/comments?post=3903"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/3903\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media\/45953"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media?parent=3903"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/categories?post=3903"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/tags?post=3903"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}