Testing private/intranet applications using Cloud-based load testing

Cloud-based Load Testing Service can be used for performance and scale testing of an application by generating load from Azure. This type of load generation can only hit/generate load on an internet/publically accessible application. But we have seen many times customer needs to load test their application which is not publically accessible. Reasons could be many, some of them are listed below:

Testing an internal application only – In many large scale organizations there are applications/web sites that have to cater the need of whole organization. It becomes crucial to test it with peak load to eradicate any performance/stress related bugs.
Testing the application internally before releasing it over internet – Before actually going into public for a big gung-ho launch, organizations love to make sure that there is no performance glitch or the worst site will not crash in high user load.

To provide support for above scenarios we are working on a feature using which users can load test their internal/ABF (application behind firewall) applications. Before talking about the solution, let us walk you through the following flow chart to figure out the best tailored solution as per your requirements. You can then read about a particular solution (1-6) and try it out.

Use default path, CLT will auto provision agents. This is the default scenario for load testing using CLT where the application has a publicly available end point. The load testing service will automatically provision load agents (in Azure) to simulate the user load. Refer this page for more details on CLT usage.
Use ARM template to deploy your own load agents. #1 solution provisions the agents in CLT service boundaries and user won’t be able to have control over it. If you wish to have control (access) over the load generation agents for any customization needs, you can deploy them yourself using ARM template in your Azure subscription. These machines will get registered with CLT can generate load. More details on this will follow in the same blog.
Use ARM template to deploy load agents in a VNet. If the application under test (AuT) is inside a Azure VNet or if there is an ExpressRoute between application’s private network and Azure, you can use a pre-defined ARM template deploy IaaS VMs in Azure in a specific VNet to act as load agents. The machines will be provisioned in your Azure subscription and registered against your VSTS account. The VNet where you create these machines should have a line of sight to the application, so that the load generators are able to reach the app. More details on this can be seen in the further segments of this blog.
Use ARM template to deploy load agents with static IPs. If you don’t have ExpressRoute connectivity and want to test apps hosted on-premises, you can use an ARM template to deploy IaaS VMs in Azure to act as load agents. Choose to create these VMs with static IPs that you can allow traffic from in the firewall, so that the necessary load can be generated. The machines will be provisioned in your Azure subscription and registered against your VSTS account. More details on this will follow in the same blog.
Use cloud load agents on your infrastructure. We have also come up with a simple powershell script that can help you to configure physical or virtual machine(s) of your choice as load agents. These machine(s) will be registered against your VSTS account for load generation. You can read more on this by following a separate blog, ‘Use cloud load agents on your infrastructure’.
Use Test Controller/Test Agents for on-premises testing on your infrastructure. If you want to do test apps on-premises, but have constraints such as not being able to store results on the cloud for some reason (say, regulatory compliance) you can use the Test Controller / Test Agents based solution for load testing. This requires you to use your own infrastructure for load generation and results will be stored on SQL Server. https://msdn.microsoft.com/en-us/library/ms243155.aspx

Now we’ll deep dive into solution #2 , #3 and #4 in this blog.

This section talks about how you can provision load agent(s) using Azure IaaS VMs in details. User must have an Azure subscription where the IaaS VMs and related resources will be provisioned. This is primarily useful in following two use cases:

You want to test a private application which is not accessible through CLT/internet.
Bring your own Subscription (BYOS) – You have your own Azure subscription and want to leverage it for load testing. You can also use the Azure free credits, if you have.

Azure gives an edge here as users can spread their load testing across different geo-locations which we have seen many customers are interesting in these days.

Use ARM template to deploy load agents in a VNet

Following is the simple topology where load agents are present under user’s VNet and henceforth they will have a line of sight to the application. We have published one ARM template in github to help user to provision machines easily and quickly.

For this to work user must have an existing VNet as depicted in above diagram. VNet identification requires its resource group name as well. If you wish to use an existing subnet, you should have this info as well.

To deploy such rig we have published one ARM template. You can click on the following link to automatically load the ARM template in Azure management portal.

Provision load agents in an existing rig

Once you click on it, it will load the template in Azure portal and you’ll see following view. User can fill in the parameters and choose the subscription/resource group/location as per the requirement.

If you wish to dig deep into the template and modify as per your needs, you can check it out from following github repo:

https://github.com/Azure/azure-quickstart-templates/tree/master/201-vsts-cloudloadtest-rig-existing-vnet

Use ARM template to deploy load agents with static IPs

We have published another ARM template where user doesn’t need to have an existing VNet. This can be used for following two purposes:

If you don’t have ExpressRoute in Azure but want to do load testing using his own subscription, you can use this ARM template which just deploys a rig with its own VNet. If you need to test an private application, you can deploy the rig with static IPs (provided as an option), punch the firewall for these IP(s) to make a route for load agents.
If you want to have control over load generation agents (as CLT auto-provisioned agents can’t be accessed by the user). You can choose to have static/dynamic IPs for these VMs.

To leverage this, just click on following link to load the ARM template in Azure management portal.

Provision load agents in a new Vnet

Github repo link for this ARM template is as follows:

https://github.com/Azure/azure-quickstart-templates/tree/master/101-vsts-cloudloadtest-rig

Once you deploy the VMs, it may take 10-15 mins to have machines configured with CLT and ready for load test. The load test runs done on these agents will not be charged VUMs by the CLT service but user will incur the cost of Azure resources consumed under his subscription.

All the VMs of a resource group get registered under an agent group whose name is same as the resource group name. In order to queue the run on a particular set of agents, you must mention the agent group name while queuing the run. If you have deployed your agent(s)/resource group(s) before Dec. 14, 2016, all of the old agents are available under an agent group named as ‘default’. We recommend to configure those agents again using the latest script.

So if you want to have your agents separated based on some configuration (e.g., location, VNet, capacity) you should keep such agents in separate resource groups. This will give you an advantage of having isolated runs and easier management of machines/agents.

In ARM template we have set the machine size to ‘Standard_D4_V2’. This size machine(s) comes up with 8 CPU cores and 28 GB of memory. User can change this value at his end by editing the template. Refer this to know more about the azure machine sizes and capabilities.

How to queue a run using load agents

Using VSTS portal

Now you can queue load test runs on self-provisioned agents through VSTS portal. As shown in below UI, you have a choice to select the type of agents in the settings tab.

Using Visual Studio Enterprise IDE

We are working on making it as first class experience in our product. Till then user can queue a run on these machines by having the following context parameter in Visual Studio Load Test file.

Context parameter name – UseStaticLoadAgents

Context parameter value – true

Context parameter name – StaticAgentsGroupName

Context parameter value – <name of the agent group>

User can set the number of machines to be used for a load test run through Agent core count property present in Run Settings. In user’s own load agents scenario every core is treated as a single machine. As shown in below image, 5 machines will be used for the run.

The runs done on user’s own load agents machines are not charged. User should be able to confirm this by looking into the status messages of the run.

Before you do that, we would recommend you to go through the FAQs as well.

Managing self-provisioned agents

To manage these self-provisioned agents users can download a powershell script from the following link:

Download powershell script to list down the registered machines

After downloading it, please make sure you unblock the file.

Following are the mandatory parameters:

TeamServicesAccountName: It is the name of the VSTS account with which you want to get configured machines. Use just need to pass the account name here e.g., please put ‘xyz’ if your VSTS account url is https://xyz.visualstudio.com
PATToken : It is required for authentication. One has to first get the PAT token for the VSTS account. Follow this post to get it. The scope should be selected as ‘Load Test (read and write)’

Following are the possible operations and their necessary switch along with the example:

Get agent groups- It’ll list down all the registered agent groups with the VSTS account. Example is as follows:

.ManageVSTSCloudLoadAgent.ps1 -TeamServicesAccountName https://xyz.visualstudio.com -PATToken olxpldk2xcdfp5bt2ruemetwrmv7xyzk23l4cnnewi3rbaq -GetAgentGroups

Get agents- It’ll list down all the agents and their latest status of a particular agent group name. Example is as follows:

.ManageVSTSCloudLoadAgent.ps1 -TeamServicesAccountName https://xyz.visualstudio.com -PATToken olxpldk2xcdfp5bt2ruemetwrmv7xyzk23l4cnnewi3rbaq -AgentGroupName test -GetAgents

Delete an agent- It’ll delete the agent reference from service. The agent must be in the offline state before a user can delete it. Agent group name is mandatory. Refer the following for example:

.ManageVSTSCloudLoadAgent.ps1 -TeamServicesAccountName https://xyz.visualstudio.com -PATToken olxpldk2xcdfp5bt2ruemetwrmv7xyzk23l4cnnewi3rbaq -DeleteAgent -AgentGroupName test -AgentName dpk-param

You can find more details by doing get-help on the ManageVSTSCloudLoadAgent.ps1.

Frequently asked question –

How do the load agents communicate with CLT? The load agents will communicate with CLT using HTTPs protocol. Since these machines/VMs are inside user’s private network (Azure/on-prem), they can reach the Application under Test (AuT) easily. The results are published back to the CLT service so that the analysis can happen in similar manner as done for other type of load test runs in CLT.
How I am being charged for this? At present this feature is in preview mode and you will not incur load testing VUM charges for the runs where you deploy load agents on your premises or in your Azure subscription. However, you will be charged the applicable Azure VM costs.
Can I use these machines for some other purpose? These machines can be used other your other tasks as well but it is recommended to not have anything running while a load test run is in progress.
Can I shut down the machines where I have configured load test agent? Yes, the machines can be shut down when not in use. The load agent service will automatically start to receive commands from CLT, once the machine is up. If you are using Azure ARM template to deploy these agents, you can start/stop the VMs based on your need. You can also do this using a powershell script. Refer following link to more on this.https://gallery.technet.microsoft.com/scriptcenter/Stop-All-VMs-in-Specified-40c8531eYou are also recommended to delete the Azure resource group once you are done with load testing. You should be able to re-create it anytime later if you need to.https://azure.microsoft.com/en-in/documentation/articles/resource-group-portal/
I have proxy settings on my machines, will this work? We support only default proxy scenario i.e., when the proxy settings are controlled through IE and it uses the current user’s credentials to connect to proxy server. In other cases, please reach out to us.
From where I can collect the logs to debug some issue? The powershell script logs are stored in the logs folder which will be present in the same folder where the powershell script exists. These logs are displayed in the powershell window as well. Run execution logs reside in %windir%tempCloudLoadTest logs folder.
How can I know the outgoing URLs so that I can whitelist them in my firewall settings if I need to. We have added one REST API for this. You first need to get the target AgentGroup Id using REST API “https://<VSTS account name>.vsclt.visualstudio.com/_apis/clt/agentgroups“. Once you get the Agent Group Id, it can be used to get the list of outgoing URLs. The syntax of the API is “https://<VSTS account name>.vsclt.visualstudio.com/_apis/clt/agentgroups?agentGroupId=<Agent Group Id>&outgoingRequestUrls=true”. The output of this will be a list of string where first two inputs are the Azure Blob/Table service URLs. Another outgoing URL is your Visual Studio account URL. Apart from these three URLs you also need to whitelist this URL “https://<VSTS account name>.vsclt.visualstudio.com.