{"id":11291,"date":"2016-02-08T21:18:18","date_gmt":"2016-02-08T21:18:18","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/?p=11291"},"modified":"2019-02-14T17:36:21","modified_gmt":"2019-02-15T01:36:21","slug":"parallel-and-context-sensitive-test-execution-with-visual-studio-2015-update-1","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/devops\/parallel-and-context-sensitive-test-execution-with-visual-studio-2015-update-1\/","title":{"rendered":"Parallel and Context Sensitive Test Execution with Visual studio 2015 Update 1"},"content":{"rendered":"

(Editors Note:\u00a0 One of the most popular series of blog posts on the ALM Blog was Terje\u2019s<\/a> posts on Unit Testing.\u00a0 So when he asked to republish his Norwegian post<\/a> on the ALM blog the answer was of course an enthusiastic YES!<\/strong>.\u00a0 A little off topic this post also highlights we need a post on executing parallel test runs with the new \u201cRun Functional Test Task\u201d as that paradigm is different using separate machines and \u201ctest containers\u201d (think assemblies))<\/p>\n

 <\/p>\n

Visual studio 2015 Update 1 : Parallel and Context Sensitive Test Execution<\/h2>\n

Visual Studio 2015 Update 1 contains a bunch of improvements and bug fixes.\u00a0 In\u00a0 this post we will have a look at two of the performance improvements, that may give a significant impact on your overall\u00a0 development performance.<\/p>\n

Parallel test execution<\/h2>\n

In Visual Studio 2015 Update 1 the test engine can execute test assemblies in parallel.\u00a0\u00a0 This can improve your test performance significantly.\u00a0 This is off by default, so you need to enable it.<\/p>\n

The system runs the test assemblies in parallel on as many cores as you specify, and of course, up to the maximum number of cores on your machine.\u00a0 If you only have a single test project you will of course have no benefit of this.\u00a0 This is a solution for the slightly larger projects.<\/p>\n

It is independent of the type of test framework you use, and therefore works with both MSTest, NUnit 2<\/a>,\u00a0 NUnit 3<\/a> and XUnit<\/a>.<\/p>\n

This mechanism is completely independent on whether the underlying framework supports parallelism or not.<\/p>\n

To turn on the feature, you must use a runsettings file, which you must enable in Visual Studio.\u00a0 The easy way to get a correct runsettings file in, is to use\u00a0 one of these templates<\/a>, note that you need at least version 3.1 of these.\u00a0 When you have this item extension installed, you have 3 templates, and if you\u2019re just after the parallelism, use the one named Parallel.<\/p>\n

Do as follow:<\/p>\n

Select Solution, Right click and choose Add\/New item<\/p>\n

You will now see a list like the one below:<\/p>\n

\"\"<\/p>\n

Select Parallel and you will see an instance of it under Solution Items in your solution.<\/p>\n

\"image_thumb15\"<\/a><\/p>\n

That\u2019s all it takes to enable parallel execution !<\/p>\n

Just remember to activate it under the Visual Studio Test menu, do like this:<\/p>\n

\"\"<\/p>\n

(1)\u00a0 Go to\u00a0 Select Test Settings file from the top menu\u00a0 Test\/Test settings.<\/p>\n

(2)\u00a0 Then select the wanted Runsettings file,\u00a0 in this case I named it Parallel2.runsettings.<\/p>\n

(3) Verify on the same menu that it is selected and checked.<\/p>\n

 <\/p>\n

The parameters:<\/h5>\n

Looking at the content of the runsettings file, we see this:<\/p>\n

\"image_thumb5\"<\/a><\/p>\n

There is only one item here we can change, the MaxCpuCount value.\u00a0 It controls how many parallel processes we can run.\u00a0 The value 0 means \u201crun as much as you can, limited by availability of cores on your machine\u201d.\u00a0 The value 1 means it will run sequentially, that is the same as the default with only one process activated.\u00a0 Any other number will limit at that number, or the number of available cores on your machine.<\/p>\n

Why don\u2019t we include Code Coverage?<\/h5>\n

You might have noticed that in the description for Parallel it says code coverage is not included. This means that when you activate this runsetting, you will not active code coverage during the same run.\u00a0 The other two templates do exactly that, it doesn\u2019t really give you any benefits.<\/p>\n

If you activate the Code Coverage the performance will be reduced, because code coverage also needs to run, and it runs sequentially after the test runs, and will therefore increase the overall time significantly.\u00a0 And, it will not give you anything you really need.<\/p>\n

The fact that you don\u2019t active Code Coverage does not imply that you can\u2019t run code coverage from Team Explorer \u2013 you can!\u00a0 Further, the results from this code coverage run is not used in the Test Explorer \u2013 strangely enough.\u00a0\u00a0 The activation done in the two other templates just activate a background code coverage run, which results are just stored in a code coverage result file under the Test Results folder.<\/p>\n

\"\"<\/p>\n

This file doesn\u2019t really give you any benefits in Visual Studio, but is useful if you want these coverage data into another program , one good example is NDepend<\/a>.<\/p>\n

One benefit is that this coverage section not only enables the background run of code coverage, but also sets up the filtering of the code coverage data<\/a>, and this might be interesting for you. These filters will also take effect when you run the Test Explorer \u201cAnalyze code coverage from selected tests\u201d<\/p>\n

In the linked article above there is a bunch of settings for code coverage, instead of using these, just use the template CompleteRunSettings, which includes this, plus the other settings.\u00a0 The XML in the linked article is not up to date.<\/p>\n

\"image_thumb9\"<\/a><\/p>\n

The templaten CompleteRunsettings has the MaxCpuCount set to 0, it will then run as parallel as possible.<\/p>\n

The templaten CoverageNoParallel is nearly identical, with exactly the same fields and settings, except the MaxCpuCount\u00a0 which is set to 1, thus sequential test running.<\/p>\n

There is also an MSDN artikkel<\/a> about configuration of unit tests which are more correct,m but it doesn\u2019t contain bthe same detailed information as the link above.\u00a0 In this however, all fields are explained.<\/p>\n

Test project for parallel runs<\/h2>\n

I have made a simple test project for parallel runs.\u00a0\u00a0 It has 4 test projects, with one test each, and each test runs for 5 seconds.\u00a0\u00a0 You can download the source code\u00a0 from here<\/a>, and then run the tests yourself.<\/p>\n

\"SNAGHTML4c90bb3b_thumb3\"<\/a><\/p>\n

(1)\u00a0 Running without settings at all. It is then runs sequentially, and we see the total time is 26 seconds.\u00a0 Since each test uses 5 seconds we have a 6 seconds overhead.<\/p>\n

(2) Running with CodeCoverageNoParallel, which is identical to the old runsettings file, we will, get code coverage run in addition, but no parallel runs.\u00a0 The total time increases to 31 seconds.<\/p>\n

(3) We then uses the complete settings, using both CodeCoverage and Parallel.\u00a0 The total time then decreases to 18 seconds.<\/p>\n

(4) And finally using only Parallel without code coverage, we\u2019re down in 12 seconds.<\/p>\n

So from (4) to (1), we gain more than a 2:1 in performance.\u00a0 That is pretty good!<\/p>\n

The machine this was tested on has more cores \u2013 8 in all, but it also runs a bunch of other programs, so not all cores can be made available for the testing processes.\u00a0 You will gain some, but not equal to the number of cores \u2013 in practice.\u00a0 Anyway, a gain of 2:1 is pretty good and pretty significant in your daily work.<\/p>\n

Parallel on build server<\/h5>\n

Does this work on a build server?\u00a0 The answer is Yes, in a build definition you can define which runsettingfile to run, so this will also have an effect on the build server run.\u00a0 But, do you really want that?\u00a0\u00a0 That all depends on how you many build servers you have, how they are configured, and how many check-ins\/commits you have.\u00a0\u00a0 That si, how loaded your build servers are.\u00a0 Normally you set up a build agent to run one per core on the machine.\u00a0 If you have many build definitions and many commits\/Checkins happening continuously during the day, the build agents will be pretty busy., then there will be little benefit in setting up the tests to run in parallel in addition, as they will \u201csteal\u201d cores from the other builds.\u00a0\u00a0 There can also be an overhead that eat up all your gain.<\/p>\n

If you, on the other hand, have a builds machine that is not that much loaded, and not so many commits continuously, then by all means \u2013 why not?\u00a0 Grab the extra performance by running your tests in parallel.<\/p>\n

There are also other ways to run in parallel, see this article<\/a> for a different approach.<\/p>\n

Context sensitive test execution<\/h2>\n

Context sensitive test execution is new in Update 1 \u2013 so what is that?\u00a0\u00a0 Under given circumstances, the new Test Explorer will only test the assemblies that it detects tests code that you have changed.<\/p>\n

This means that if you work with a certain piece of code in a given module of your system,\u00a0 and your code is distributed over multiple modules, then for each build, only the code that you changes is being build. That is the normal incremental build which is by default.\u00a0 New is that also only the tests touching this code is being run, and then not the tests for code that has not been touched.\u00a0 It is pretty obvious that for larger systems this can mean a pretty dramatic speedup in the test run performance.<\/p>\n

There is one condition for this to work:\u00a0 You must activate \u201cRun tests after build\u201d.<\/p>\n

You do this either in the Test Explorer (1) or from the top menu Test\/Test Settings\/Run Tests after Build (2).<\/p>\n

\"SNAGHTML4cb5284e_thumb2\"<\/a><\/p>\n

The effect of this can be seen in the screenshots below:<\/p>\n

\"image_thumb14\"<\/a><\/p>\n

(1) Activate \u201cRun tests after build\u201d<\/p>\n

(2) Do some code changes<\/p>\n

(3) Just select Build,\u00a0 or use the keyboard shortcut<\/p>\n

(4) Just the touched test are being run, the other ones are shown dimmed green.\u00a0\u00a0 They ran green the last time they were run, but they were not run this time, because no code what changed that affected them. .<\/p>\n

If you have larger projects this can give a significant gain in performance, and make your turn around from build through test back to edit mode much faster \u2013 and that is what is important during development, right?<\/p>\n","protected":false},"excerpt":{"rendered":"

(Editors Note:\u00a0 One of the most popular series of blog posts on the ALM Blog was Terje\u2019s posts on Unit Testing.\u00a0 So when he asked to republish his Norwegian post on the ALM blog the answer was of course an enthusiastic YES!.\u00a0 A little off topic this post also highlights we need a post on […]<\/p>\n","protected":false},"author":81,"featured_media":45953,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1,252],"tags":[],"class_list":["post-11291","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","category-testing"],"acf":[],"blog_post_summary":"

(Editors Note:\u00a0 One of the most popular series of blog posts on the ALM Blog was Terje\u2019s posts on Unit Testing.\u00a0 So when he asked to republish his Norwegian post on the ALM blog the answer was of course an enthusiastic YES!.\u00a0 A little off topic this post also highlights we need a post on […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/11291","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/users\/81"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/comments?post=11291"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/11291\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media\/45953"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media?parent=11291"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/categories?post=11291"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/tags?post=11291"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}