{"id":29056,"date":"2017-04-04T13:03:45","date_gmt":"2017-04-04T20:03:45","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/visualstudioalm\/?p=29056"},"modified":"2019-02-14T15:55:40","modified_gmt":"2019-02-14T23:55:40","slug":"integrating-smoke-tests-into-your-continuous-delivery-pipeline","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/devops\/integrating-smoke-tests-into-your-continuous-delivery-pipeline\/","title":{"rendered":"Integrating Smoke Tests into your Continuous Delivery Pipeline"},"content":{"rendered":"<blockquote><p><em>We&#8217;re really glad to have <a href=\"https:\/\/twitter.com\/AbelSquidHead\">Abel Wang<\/a> help us out for\u00a0<a href=\"https:\/\/aka.ms\/SpringIntoDevOps\">#SpringIntoDevOps<\/a> with this awesome blog contribution about verifying whether your deployment finished successfully by integrating smoke tests into your pipeline. \u00a0Thank you Abel! \u00a0&#8212; <a href=\"https:\/\/twitter.com\/edblankenship\">Ed Blankenship<\/a><\/em><\/p><\/blockquote>\n<hr \/>\n<p><span><\/span>Having a Continuous Integration (CI) and Continuous Delivery (CD) pipeline in Visual Studio Team Services enables us to build and release our software quickly and easily.\u00a0 Because of the high volume of builds and releases that can occur, there is a chance that some of the releases will fail.\u00a0 Finding these failures early is vital.\u00a0 Using integrated smoke tests in your CD pipeline is a great way to surface these deployment failures automatically after each deployment.<\/p>\n<p>There are two types of smoke tests you can run.\u00a0 Functional tests where you write code that tests which verifies your app is deployed and working correctly, and automated UI tests that will actually exercise the user interface using automated user interface test scripts.\u00a0 Both these types of smoke tests can be run in your CD pipeline using the Visual Studio Test task.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/1.png\" alt=\"1\" width=\"472\" height=\"484\" class=\"aligncenter size-full wp-image-29066\" \/><\/p>\n<p>The <a href=\"https:\/\/www.visualstudio.com\/en-us\/docs\/test\/continuous-testing\/getting-started\/getting-started-with-continuous-testing\">Visual Studio Test Task<\/a> can run tests using multiple testing frameworks including MSTest, NUnit, xUnit, Mocha and Jasmine.\u00a0 The task actually uses <em>vstest.console.exe<\/em> to execute the tests.\u00a0 For this blog post, I\u2019ll be using MSTest, but you can use whatever testing framework you want with the correct test adapter.<\/p>\n<p>Using MSTest, it\u2019s very simple to create smoke tests.\u00a0 At the end of the day, tests in MSTest (or any of the other testing frameworks) are just chunks of code that is run.\u00a0 Anything you can do with code can be part of your smoke tests.\u00a0 Some common scenarios include:<\/p>\n<ul>\n<li>hitting a database and making sure it has the correct schema,<\/li>\n<li>checking if certain data is in the database,<\/li>\n<li>hitting a service and making sure the response is correct,<\/li>\n<li>hitting URLs and making sure some dynamic content is returned back<\/li>\n<\/ul>\n<p>Automated UI tests can also be done using MSTest (or another testing framework) with Selenium or Coded UI or whatever automation technology you want to use.\u00a0 Remember, if you can do it with code (in this case C#) then you can get the Visual Studio Test task to do it.\u00a0 For this blog, we will be looking at creating automated UI smoke tests.<\/p>\n<p>The first thing we need to do is make sure your smoke test project is part of the solution that gets compiled in your automated build.\u00a0 In this example, I have a solution that includes a\u00a0web project, a MSTest project used for my smoke tests and some other projects.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/2.png\" alt=\"2\" width=\"315\" height=\"484\" class=\"aligncenter size-full wp-image-29075\" \/><\/p>\n<p>For the automation scripts, I used Selenium with the Page Object pattern where I have an object which represents each page of my app and each page object has all the actions you can do on the page as well as the asserts that you can do on the page.\u00a0 This creates some super clean smoke tests.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/3.png\" alt=\"3\" width=\"644\" height=\"245\" class=\"aligncenter size-full wp-image-29085\" \/> <img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/4.png\" alt=\"4\" width=\"644\" height=\"282\" class=\"aligncenter size-large wp-image-29095\" \/> <img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/5.png\" alt=\"5\" width=\"644\" height=\"263\" class=\"aligncenter size-large wp-image-29105\" \/><\/p>\n<p>Make sure your build compiles your test project and the test project\u2019s <em>.dll<\/em> is one of the build artifacts.\u00a0 For this example, I set up my release steps to do the following:<\/p>\n<ol>\n<li>Deploy web app<\/li>\n<li>Deploy database schema changes<\/li>\n<li>Massage my configuration files so my Selenium tests hits my Dev environment and uses Google Chrome as the browser<\/li>\n<li>Add release annotations for Application Insights<\/li>\n<li>Run Selenium Tests<\/li>\n<\/ol>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/6.png\" alt=\"6\" width=\"360\" height=\"272\" class=\"aligncenter size-full wp-image-29115\" \/><\/p>\n<p>Setting up the Visual Studio Test task in the CD pipeline to run my automated UI smoke tests using MSTest is straight forward.\u00a0 All it requires is setting some parameters.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/7.png\" alt=\"7\" width=\"367\" height=\"484\" class=\"aligncenter size-full wp-image-29125\" \/><\/p>\n<p>For detailed descriptions on all the parameters you can set for the Visual Studio Test task, check out <a href=\"https:\/\/github.com\/Microsoft\/vsts-tasks\/blob\/releases\/m109\/Tasks\/VsTest\/README.md\">https:\/\/github.com\/Microsoft\/vsts-tasks\/blob\/releases\/m109\/Tasks\/VsTest\/README.md<\/a>.\u00a0 In my example, I am running tests contained in any <em>.dll<\/em> that has the word test in it.\u00a0 I\u2019m also filtering and only running tests that are in the <em>TestCategory=UITests<\/em>.\u00a0 You have lots of options for how you want to categorize and structure your tests.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/8.png\" alt=\"8\" width=\"351\" height=\"484\" class=\"aligncenter size-full wp-image-29135\" \/><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/9.png\" alt=\"9\" width=\"644\" height=\"245\" class=\"aligncenter size-full wp-image-29136\" \/><\/p>\n<h2>Automated User Interface Smoke Tests<\/h2>\n<p>Automated UI tests require a private VSTS build\/deploy agent running in interactive mode. If you have not ever setup an agent to run interactively, there is a <a href=\"https:\/\/www.visualstudio.com\/en-us\/docs\/build\/concepts\/agents\/agents\">great walkthrough for installing and configuring agent for interactive mode<\/a>.\u00a0 Alternatively, you can actually run these same smoke tests using <em>phantomjs<\/em> (headless) which will work with the hosted agents in VSTS.\u00a0 To run my smoke tests using <em>phantomjs<\/em>, just change the environment variable <em>Token.BrowserType<\/em> from <em>chrome<\/em> to <em>phantomjs<\/em>.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/10.png\" alt=\"10\" width=\"644\" height=\"414\" class=\"aligncenter size-full wp-image-29145\" \/><\/p>\n<p>Now, when a release is triggered, after deploying my web app and database, I run my set of smoke tests using the Visual Studio Test task and all results are automatically posted back to the release summary and to VSTS.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/11.png\" alt=\"11\" width=\"644\" height=\"414\" class=\"aligncenter size-full wp-image-29155\" \/> <img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/12.png\" alt=\"12\" width=\"644\" height=\"414\" class=\"aligncenter size-large wp-image-29165\" \/><\/p>\n<h2>Smoke Tests for Mobile Continuous Delivery<\/h2>\n<p>The Continuous Delivery system in VSTS is so flexible, we can even configure it to run smoke tests in complex mobile scenarios.\u00a0 In the following example, I have an app that consists of a website, multiple REST API services, a back-end database and a mobile app.\u00a0 My release pipeline consists of:<\/p>\n<ul>\n<li>Create or update my Azure Resource Group from an ARM template<\/li>\n<li>Deploy Web App and Services<\/li>\n<li>Deploy database schema changes<\/li>\n<li>Do a web performance test<\/li>\n<li>Run some Selenium Automated UI tests for smoke tests against the web site and services<\/li>\n<li>Deploy my mobile app to my Dev Testers group using HockeyApp<\/li>\n<li>Deploy and run my smoke tests against my mobile app using Xamarin.UITests in Xamarin Test Cloud using the Xamarin Test Cloud Task<\/li>\n<\/ul>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/devops\/wp-content\/uploads\/sites\/6\/2017\/04\/13.png\" alt=\"13\" width=\"644\" height=\"353\" class=\"aligncenter size-full wp-image-29175\" \/><\/p>\n<p>Using smoke tests as part of your CD pipeline is a valuable tool to help ensure your deployment, configuration and resources are all working.\u00a0 Release Management\u00a0in Visual Studio Team Services is fully configurable and customizable to run any type of smoke tests that you want as part of the deployment steps.\u00a0 The source code for the examples in this blog are in GitHub\u00a0 <a href=\"https:\/\/github.com\/abelsquidhead\/SmokeTestsForCD.git\">here<\/a>.<\/p>\n<p>&nbsp;<\/p>\n<p><a href=\"https:\/\/twitter.com\/AbelSquidHead\">Abel Wang<\/a>\nSenior Technical Product Marketing Manager, Visual Studio Team Services<\/p>\n","protected":false},"excerpt":{"rendered":"<p>We&#8217;re really glad to have Abel Wang help us out for\u00a0#SpringIntoDevOps with this awesome blog contribution about verifying whether your deployment finished successfully by integrating smoke tests into your pipeline. \u00a0Thank you Abel! \u00a0&#8212; Ed Blankenship Having a Continuous Integration (CI) and Continuous Delivery (CD) pipeline in Visual Studio Team Services enables us to build [&hellip;]<\/p>\n","protected":false},"author":105,"featured_media":45953,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-29056","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"acf":[],"blog_post_summary":"<p>We&#8217;re really glad to have Abel Wang help us out for\u00a0#SpringIntoDevOps with this awesome blog contribution about verifying whether your deployment finished successfully by integrating smoke tests into your pipeline. \u00a0Thank you Abel! \u00a0&#8212; Ed Blankenship Having a Continuous Integration (CI) and Continuous Delivery (CD) pipeline in Visual Studio Team Services enables us to build [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/29056","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/users\/105"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/comments?post=29056"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/29056\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media\/45953"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media?parent=29056"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/categories?post=29056"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/tags?post=29056"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}