Testing in Continuous Integration and Continuous Deployment Workflows

Gopinath C H[MSFT]

May 29th, 20151 0

When Brian announced the release of Visual Studio 2015 RC and Team Foundation Server 2015 RC, he blogged about lots of features in these releases including Build.Preview and Release Management which are key for DevOps.

At a high level DevOps is to focus on increasing customer value by reducing cycle times in the development process. There are lots of ways to achieve this – increase developer productivity, speed the feedback process, and eliminate friction in the testing and deployment process. This post is focused on this latter tactic, while this post will be looking at testing in continuous integration using Build.Preview in Visual Studio Online the same tasks can be used in a continuous deployment (or continuous delivery) workflow with Release Management as well.

If you are not familiar with continuous integration or continuous deployment it is where the build, unit tests, acceptance tests and product release all happen automatically in a workflow. In continuous integration, focus is getting an automated good build out with basic build verification tests while continuous deployment is focused heavily on testing and release management flows.

A typical continuous integration workflow will contain:

Setting up the automated build and unit testing
Setting up machines for application deployment and testing
Configuring for application deployment and testing
Queuing the Build, execute Tests and Test results analysis
Configuring the whole work flow to execute on a continuous basis

We have used the following principles while developing the above workflow:

Open and Extensible
- Build framework of your choice – you can use MSBuild, Maven, Ant, Gradle etc.
- Test Frameworks of your choice – you can run MSTest, Selenium Tests, Junit, Nunit etc.
- Task model is fully extensible and you can add your own tasks.
Reusing assets
- We have designed it such a way that the assets/tasks you create can be used on both continuous integration and release management workflow
Simplified machine provisioning and deploying experience
Distributed Test Execution
- Effectively execute a large number of diverse automated tests (Unit, Functional UI – Selenium and Coded UI, Functional Non-UI, 3rd party adapter, Cloud Load Tests etc.) on remote machine group
Rich Test reports and Test Results Analysis
- Drill down into test results and analyze test failures, helping you validate the quality of a build or a release.

The new Test tools in Visual Studio Online and Team Foundation Server 2015 make it easy to achieve the Continuous Integration Automation. You can see the quick video which shows Continuous Integration workflow. Let us go through in detail on each of those steps, following sections focus on the above workflow and how it can be automated in Visual Studio Online.

For this walkthrough we assume that you have a simple web application, unit tests and automated selenium tests all working and checked into visual studio online git/tfvc. For simplicity of the workflow we also assumed that this web application is an enterprise application and will be hosted inside an enterprise and it does not have any other dependencies like database server or middle tier etc.

We are still working on these features, some of these experiences including UI will change as we improve and also these features may not show up today but will show up as we are deploy and enable it. Please note that they are still under development and should not be used in production.

Please share your feedback below in comments or mail us your feedback.

Setup Build and Unit Test Tasks

As it was announced earlier Build.Preview can be used to create automated builds and it is the main hub for continuous integration workflow.

Prerequisites for executing build definitions is to have your build agent ready, here are steps to setup your build agent, you can find more details in this blog .

1) First step is create a build definition and select “Visual Studio” template.

Figure 1Build-Build Definition

2) Selecting Visual Studio template will automatically add a Build Task and Unit Test Task. Please fill in the parameters needed for each of the tasks. Build task is straight-forward, it just takes the solution that has to be built and the configuration parameters. As I had mentioned earlier this solution contains product code, unit test code and also automated selenium tests that we want to run as part of build validation.

Figure 2Build-Build Task

3) Final step is to add the required parameters needed for the Unit Test task – Test Assembly and Test Filter criteria. One key thing you notice below in this task is we take the unit test dll and enumerate all tests in it and run the tests automatically. You can include a test filter criteria and filter on traits defined in test cases if you want to execute specific tests. Another important point, unit tests in Visual Studio Test Task always run on build machine and do not require any deployment/additional setup. See figure 3 below.

Figure 3 Build-Unit Test Task

Setting up machines for application deployment and running tests

Once the Build is done and the Unit tests have passed, the next step is to deploy the application (website) and run functional tests.

Prerequisites for this are:

Already provisioned and configured Windows Server 2012 R2 with IIS to deploy the web site.
A set of machines with all browsers (Chrome, Firefox and IE) installed to automatically run Selenium tests on these machines.

Please make sure Powershell Remote is enabled on all the machines.

Once the machines are ready, go to the Test Hub->Machine page to create the required machine configuration as shown in the below screen shots.

Enter machine group name and enter the FQDN/IP Address of the IIS/Web Server machine that is setup earlier. You might also need to enter the admin username and password for the machine for all further configurations. The application under test environment, should always be a test environment not a production environment as we are doing integration tests targeting the build.

Figure 4 Machine Hub

For Test Environment, give a name to the test environment and add all IP address of the lab machines that were setup already with the browsers. As I had mentioned earlier test automation system is capable of executing all tests in a distributed way and can scale up to any number of machines (we will have another blog).

At the end of this step, in machines hub you should have one application under test environment and a test environment, in the example we are using them as “Application Under Test” and “Test Machines” respectively as the names of the machine groups.

Configuring for application deployment and testing

In this section, we will show you how to add deployment task for deploying the application to the web server and remote test execution tasks to execute integration tests on remote machines.

We will use the same build definition and enhance it to add the following steps for continuous integration:

1) Deploying the Web Site using Powershell: We first need to copy the all the website files to the destination. Click on “Add build step” and add “Windows Machine File Copy” task and fill in the required details for copying the files. Then add “Run Powershell on Target Machine Tasks” to the definition for deploying/configuring the Application environment. Chose “Application under Test” as the machine group that we had setup earlier for deploying the web application to the web server. Choose powershell script for deploying the website (if you do not have deployment web project, create it). Please make sure to include this script in the solution/project. This task executes powershell script on the remote machine for setting up the web site and any additional steps needed for the website.

Figure 5 Build – Application Deploy Powershell Task

2) Copy Test Code to the Test Machines: As Selenium UI tests which are we are going to use as integration tests are also built as part of the build task, add “Copy Files” task to the definition to copy all the test files to the test machine group “Test Machines” which was configured earlier. You can chose any test destination directory in the below example it is “C:Tests”

Figure 6Build – Copy Files Task

3) Deploy Visual Studio Test Agent: To execute on remote machines, you first deploy and configure the test agent. To do that, all you need is a task where you supply the remote machine information. Setting up lab machines is as easy just adding a single task to the work flow. This task will deploy “Test Agent” to all the machines and configures them automatically for the automation run. If the agent is already available and configured on the machines, this task will be a no-op.

Unlike older versions of Visual Studio – now you don’t need to go manually copy and setup the test controller and test agents on all the lab machines. This is a significant improvement as all the tasks can be done remotely and easily.

Figure 7Build – Deploy Test Agent Task

4) Run Tests on the remote Machines: Now that the entire lab setup is complete, last task is to add “Run Visual Studio Tests using Test Agent” task (renamed to “Run Functional Tests” task from TFS 2015 Update 2) to actually run the tests. In this task specify the Test Assembly information and a test filter criteria to execute the tests. As part of build verification we want to run only P0 Selenium Tests, so we will filter the assemblies using SeleniumTests*.dll as the test assembly.

You can include a runsettings file with your tests and any test run parameters as input. In the example below, we are passing the deployment location of the app to the tests using the $(addurl) variable.

Figure 8Build – Run Tests Task

Once all tasks are added and configured save the build definition.

Queue the build, execute tests and test run analysis

Now that the entire set of tasks are configured, you can verify the run by queuing the build definition. Before queuing the build make sure that the build machine and test machine pool is setup.

Once the build definition execution is complete, you will get a great build summary with all the required information needed for you to take the next steps.

As per the scenario, we have completed the build, executed unit tests and also ran Selenium Integration Tests on remote machines targeting different browsers. Let’s say we got few failures in our integration tests, as a result, the build is shown as partially succeeded.

Build Summary has the following information:

A summary of steps that have passed and color coded on the left and details in the right side panel.
You can click on each task to see detailed logs.
From the tests results, you can see that all unit tests passed and there were failures in the integration tests.

Figure 9Build – Result Summary

Next step is to drill down and understand the failures. You can simply click on the Test Results link in the build summary to navigate to the test run results.

Based on the feedback, we have created a great Test Run summary page with a set of default charts and also mechanism to drill down into the results. Default summary page has the following built-in charts readily available for you – Overall Tests Pass/Fail, Tests by priority, configuration, failure type etc.

Figure 10Run – Test Run Summary

If you want to drill deeper on the tests, you can click on the “Test Results” tab, you will get to see each and every test – test title, configuration, owner, machine where it was executed etc.

For each failed test, you can click on the “Update Analysis” to analyze the test. In the below summary you notice that IE Selenium tests are failing. You can directly click on “Create Bug” link at the top to file bugs, it will automatically take all test related information metadata from the results and include it in the bug – it is so convenient.

Figure 11Run – Test Results

Configuring for Continuous Integration

Now that the tests are all investigated and bugs filed, you can configure the above build definition for Continuous Integration to run build, unit tests and key integration tests automatically for every subsequent check-in. You can navigate to the build definition and click on Triggers.

You have two ways to configure:

Select “Continuous Integration” to execute the workflow for all batched check-ins
Select a specific schedule for validating the quality after all changes are done.

You can also choose both as shown below, the daily scheduled drop can be used as daily build for other subsequent validations and consuming it for partner requirements.

Figure 12Build – Continuous Integration

Using the above definition, you are now setup for “Continuous Integration” of the product to automatically build, run unit tests and also key integration tests for validating the builds. All the tasks shown above can be used in Release Management workflow as well to enable Continuous Delivery scenarios.

Summary

To summarize what all we have achieved in this walk through:

Created a simple build definition with build, unit testing and automated tests
Simplified experience in configuring machines and test agents
Improvements in build summary and test runs analysis
Configuring the build definition as a “continuous integration” for all check-ins