{"id":19236,"date":"2016-07-25T16:23:23","date_gmt":"2016-07-25T23:23:23","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/?p=19236"},"modified":"2019-02-14T17:33:48","modified_gmt":"2019-02-15T01:33:48","slug":"evolving-the-visual-studio-test-platform-part-1","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/devops\/evolving-the-visual-studio-test-platform-part-1\/","title":{"rendered":"Evolving the Visual Studio Test Platform &#8211; Part 1"},"content":{"rendered":"<p>Three releases (<a href=\"https:\/\/blogs.msdn.microsoft.com\/visualstudio\/2016\/06\/27\/visual-studio-2015-update-3-and-net-core-1-0-available-now\/\">VS 2015 Update 3<\/a>, <a href=\"https:\/\/blogs.msdn.microsoft.com\/visualstudio\/2016\/07\/07\/visual-studio-15-preview-3\/\">Visual Studio &#8220;15&#8221; Preview 3<\/a>, <a href=\"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/2016\/06\/17\/taking-the-mstest-framework-forward-with-mstest-v2\/\">MSTest V2<\/a>) featuring Test Platform components in as many months indicate a path best traced by starting from the present.<\/p>\n<p><strong>The Test Platform<\/strong>\nPresently, Visual Studio has an open and extensible test platform with tests being written using various test frameworks and run using a <a href=\"https:\/\/visualstudiogallery.msdn.microsoft.com\/site\/search?query=%22Test%20Adapter%22&amp;f%5B0%5D.Value=%22Test%20Adapter%22&amp;f%5B0%5D.Type=SearchText&amp;ac=4\">variety of adapters<\/a>. The Test Platform, from its vantage, resolves the lifecycle of the test into a series of stages \u2013 two of which are writing and running the test\u00a0\u2013 with the goal of providing extensibility at each stage.<\/p>\n<p><strong>The Test Lifecycle<\/strong>\nThe below diagram illustrates the lifecycle:<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/6\/2019\/05\/TestLifeCycle.jpg\" rel=\"attachment wp-att-19245\"><img decoding=\"async\" width=\"2560\" height=\"1440\" class=\"alignnone size-full wp-image-19245\" alt=\"TestLifeCycle\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2016\/07\/TestLifeCycle.jpg\" \/><\/a><\/p>\n<p>And in broad terms, the stages can be described as follows:<\/p>\n<ul>\n<li>The Test Platform supports running tests written in various test frameworks using a pluggable model. Based on user-choice, the desired test framework and its corresponding adapter can be <strong><em><a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/hh598952.aspx\">acquired<\/a><\/em> <\/strong>as a vsix or as NuGet package as the case may be. Adapters can be written in terms of a public API exposed by the Test Platform &#8211; for e.g. here is a description of the <a href=\"http:\/\/matthewmanela.com\/blog\/anatomy-of-the-chutzpah-test-adapter-for-vs-2012-rc\/\">Chutzpah adapter&#8217;s implementation<\/a>.<\/li>\n<li>The next stage is to write the test<a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/hh598957.aspx\"><\/a>. Based on the framework\/adapter tuple, tests may be\u00a0<strong><em>authored <\/em><\/strong>for <a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/hh598960.aspx\">Managed code<\/a>, for <a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/hh598953.aspx\">native code<\/a>,\u00a0for <a href=\"https:\/\/github.com\/mmanela\/chutzpah\/wiki\/Running-JavaScript-tests-with-Chutzpah\">JavaScript code<\/a>, etc. When\u00a0authored\u00a0from within the Visual Studio IDE, then Project templates, wizards, IntelliSense and the rest of the code editing infrastructure comes into play here.<\/li>\n<li>All authored tests are made available for running. Optionally, various attributes of the tests may be used to <strong><em>select<\/em><\/strong> a subset. Selection can be done by test case name, by grouping them based on criteria like project\/class\/traits\/outcome\/duration, by applying a filter, etc. This ability to select tests is available from the <a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/hh270865.aspx#Anchor_4\">Test Explorer<\/a>, from the command line (<a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/jj155800(v=vs.120).aspx\">vstest.console.exe<\/a>), as well as from the <a href=\"https:\/\/github.com\/Microsoft\/vsts-tasks\/tree\/master\/Tasks\/VsTest\">VSTest task<\/a>. Selection enables focusing on the set of tests\u00a0to run.<\/li>\n<li>The system\u00a0<strong><em><a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/jj635153.aspx\">configuration<\/a><\/em><\/strong>\u00a0to use to run the tests\u00a0\u2013 the processor architecture (x86\/x64) to use, the target .NET Framework to use, the set of data collectors to be in effect when the tests are run, etc.\u00a0 \u2013 can be precisely set. Such configuration is set using the .runsettings file, and there are adapter-specific sections within this file (owned by the adapter writers, of course).<\/li>\n<li>Then comes the <strong><em><a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/hh270865.aspx\">run<\/a><\/em><\/strong> stage where the selected tests are run as configured and using the appropriate adapter. This is the testrun \u2013 the most often repeated stage in the life cycle. Tests may be run using the <a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/hh270865.aspx\">Test Explorer<\/a>,\u00a0command line (<a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/jj155800(v=vs.120).aspx\">vstest.console.exe<\/a>), as well as from the <a href=\"https:\/\/github.com\/Microsoft\/vsts-tasks\/tree\/master\/Tasks\/VsTest\">VSTest task<\/a>.<\/li>\n<li>Application-platform-specific <strong><em><a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/hh270865.aspx#BKMK_Debug_and_analyze_unit_tests\">debugging<\/a><\/em><\/strong> is supported to enable high-fidelity debugging for tests targeting the desktop, Store, UWP, and so on.<\/li>\n<li>Feedback from the testrun is <strong><em><a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/hh270865.aspx#BKMK_Debug_and_analyze_unit_tests\">analyzed<\/a><\/em><\/strong> to figure out the outcome of each test, the duration, and \u2013 if configured\u00a0\u2013 the code coverage data, and data from any other data collectors that may have been active.<\/li>\n<li>The feedback is then <strong><em><a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/ms182495(v=vs.100).aspx\">reported<\/a><\/em><\/strong> for user-consumption. This is done on the <a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/hh270865.aspx#Anchor_3\">Test Explorer<\/a> if you are in the Visual Studio IDE or \u2013\u00a0if you are using the commandline\u00a0\u2013 on stdout, as a <a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/ms182495(v=vs.100).aspx\">.trx file<\/a>, or <a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/jj155796.aspx\">published<\/a> to TFS. <a href=\"https:\/\/blogs.msdn.microsoft.com\/vikramagrawal\/2012\/07\/26\/writing-loggers-for-command-line-test-runner-vstest-console-exe\/\">Loggers<\/a>, like adapters, can be plugged-in to the Test Platform to provide custom logging\/reporting.<\/li>\n<\/ul>\n<p><strong>Efficiency as a Theme for features across the life cycle<\/strong>\nEfficient execution has been the pervading theme for the features we have implemented across the stages in this lifecycle\u00a0\u2013 not restricted only to efficient &#8220;running&#8221; of the tests alone, but efficiency in executing across the entire lifecycle. After all, as a developer, one wants to get from conceptualizing the tests, all the way to the point where they are generating reports that can inform business decisions.<\/p>\n<p><strong>Next post \u2013 recap of the features<\/strong>\nWe will continue this series in the\u00a0next post, looking at the features we have implemented and delivered in the Visual Studio 2015 cycle that enable such efficiency.<\/p>\n<p>Stay tuned.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Three releases (VS 2015 Update 3, Visual Studio &#8220;15&#8221; Preview 3, MSTest V2) featuring Test Platform components in as many months indicate a path best traced by starting from the present. The Test Platform Presently, Visual Studio has an open and extensible test platform with tests being written using various test frameworks and run using [&hellip;]<\/p>\n","protected":false},"author":765,"featured_media":45953,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[226,1,252],"tags":[],"class_list":["post-19236","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ci","category-devops","category-testing"],"acf":[],"blog_post_summary":"<p>Three releases (VS 2015 Update 3, Visual Studio &#8220;15&#8221; Preview 3, MSTest V2) featuring Test Platform components in as many months indicate a path best traced by starting from the present. The Test Platform Presently, Visual Studio has an open and extensible test platform with tests being written using various test frameworks and run using [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/19236","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/users\/765"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/comments?post=19236"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/19236\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media\/45953"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media?parent=19236"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/categories?post=19236"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/tags?post=19236"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}