SiteCore is a business intelligence company based out of Copenhagen Denmark. They provide a solution that allows developers to understand deep business analytics in their own services. We had the opportunity to work with SiteCore’s development team in Kuala Lumpur for a one-week hackfest in order to explore how they could use Application Insights, an Azure application telemetry service, to understand the health and availability of their services.

The Problem

One of the things that SiteCore does for each deployment is to run a series of recurring web tests. These are tests that ping an endpoint and expect a particular response. So for example, a test may be pseudo-coded as follows:

GET http://deploymenthost.com/tests/databaseconnection

EXPECTED RESPONSE: 200

The test will periodically call the specified endpoint with the specified method and headers and returns a response. When the endpoint is called by the web test the actual test logic is performed server-side. Sometimes this is ensuring a database is connected, other times it may just be checking to see if the service itself is still running. If the test fails, then an alert needs to be sent to the appropriate maintainer.

SiteCore’s unique problem is that they deploy a large number of services and the endpoints often change depending on new tests, host names or a combination of both. They needed a way to deploy their tests based on information known only at deploy-time.

The Solution

Application Insights offers automated web tests which can be ran from any Azure data center in the world to ensure availability for a variety of markets. You can easily create a web test from the portal:

This works great for simple scenarios, however in the case of SiteCore, the number of tests and the properties of a test are very fluid and can change per deployment. Because of this we needed to be able to describe web tests programmatically.

Applications Insights does this, but its a bit tricky at first as documentation around this is quite scarce – which hopefully this post sheds some light on.

Describing a Web Test with XML

So it turns out that Web Tests are actually described by an XML schema, which coincidentally is used by Visual Studio Web Tests.

We spared you the trouble of creating a webtest in Visual Studio and reverse-engineering the schema. Here’s what a typical web ping test XML looks like:

<WebTest Name="WebTest1" Id="0000-0000-0000-0000" Enabled="True" CssProjectStructure="" CssIteration="" Timeout="0" WorkItemIds="" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010" Description="" CredentialUserName="" CredentialPassword="" PreAuthenticate="True" Proxy="default" StopOnError="False" RecordedResultFile="" ResultsLocale="">
  <Items>
    <Request Method="GET" Guid="a5f10126-e4cd-570d-961c-cea43999a200" Version="1.1" Url="http://microsoft.com" ThinkTime="0" Timeout="300" ParseDependentRequests="True" FollowRedirects="True" RecordResult="True" Cache="False" ResponseTimeGoal="0" Encoding="utf-8" ExpectedHttpStatusCode="200" ExpectedResponseUrl="" ReportingName="" IgnoreHttpStatusCode="False" />
  </Items>
</WebTest>

It turns out that in Azure Resource Manager we can deploy App Insights web test resources which are of the resource type Microsoft.Insights/webtests.

A Microsoft.Insights/webtests resource may look like this (a referenced Microsoft.Insights/component is omitted for brevity):

{
  "name": "Test1",
  "apiVersion": "2015-05-01",
  "type": "microsoft.insights/webtests",
  "location": "Central US",
  "tags": {
    "[concat('hidden-link:', resourceId('microsoft.insights/components/appinsightsresource')))]": "Resource"
  },
  "dependsOn": [
    "[concat('microsoft.insights/components/', parameters('appName'))]"
  ],
  "properties": {
    "Name": "Test1",
    "Description": "A healthy descriptive piece of text",
    "Enabled": true,
    "Frequency": 300,
    "Timeout": 30,
    "Kind": "ping",
    "Locations": [{ "Id": "us-il-ch1-azr" }],
    "Configuration": {
      "WebTest": "<WebTest Name="WebTest1" Id="0000-0000-0000-0000" Enabled="True" CssProjectStructure="" CssIteration="" Timeout="0" WorkItemIds="" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010" Description="" CredentialUserName="" CredentialPassword="" PreAuthenticate="True" Proxy="default" StopOnError="False" RecordedResultFile="" ResultsLocale=""><Items><Request Method="GET" Guid="a5f10126-e4cd-570d-961c-cea43999a200" Version="1.1" Url="http://microsoft.com" ThinkTime="0" Timeout="300" ParseDependentRequests="True" FollowRedirects="True" RecordResult="True" Cache="False" ResponseTimeGoal="0" Encoding="utf-8" ExpectedHttpStatusCode="200" ExpectedResponseUrl="" ReportingName="" IgnoreHttpStatusCode="False" /></Items></WebTest>"
    },
    "SyntheticMonitorId": "Test1"
  }
}

After discovering this, you can tell that it would be easy to programmatically create web tests by using a template XML file and rendering that XML file with test-specific parameters such as the expected HTTP status code or the URL for the test to use.

SiteCore originally wanted to have this logic within a Powershell script they currently use to create their deployments. They would have to use the Azure Resource Manager APIs to create the resources individually and manage the deployment of each. This works, but brings additional complexity and management to the deployment logic.

It turns out there is an easier way to do this. Using Azure Resource Manager, we can leverage desired-state deployments with a bit of clever template writing.

Writing Advanced Azure Resource Management Templates

You may have heard of Azure Resource Management templates which allow you to describe infrastructure by extending JSON syntax. What you may have not known is that Azure Resource Manager Template Language comes with a variety of features to bring some basic logic such as iteration and arithmetic into a template allowing for more complex deployments with much less code.

Resource Manager templates have the ability to deploy resources iteratively using the copy resource management property. For a given resource, using the copy property you can specify the name of the copy operation as well the number of times it should iterate:

"copy": {
  "name": "createTests",
  "count": 10
}

The next question you may ask is, ‘How does the template know how many and what kind of tests to deploy’. Using a template parameters file, we can actually pass more complex template parameters than what you may see on a typical Azure Resource Manager template parameters file.

Using a parameters file which passes an array of objects, we can use each object to describe all the necessary information for each test:

{
  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "appName": {
      "value": "someapp"
    },
    "emails": {
      "value": [
        "youremail@yourdomain.com"
      ]
    },
    "tests": {
      "value": [
        {
          "name": "test1",
          "url": "http://www.microsoft.com",
          "expected": 200,
          "frequency_secs": 300,
          "timeout_secs": 30,
          "failedLocationCount": 1,
          "description": "a description for test1",
          "guid": "cc1c4b95-0a39-48ce-9c7b-fa41f0fc0bee",
          "locations": [{
            "Id": "us-il-ch1-azr"
          }]
        },
        {
          "name": "test2",
          "url": "http://www.microsoft.com",
          "expected": 404,
          "frequency_secs": 300,
          "timeout_secs": 30,
          "failedLocationCount": 1,
          "description": "a description for test3",
          "guid": "cc1c4b95-0a39-48ce-9c7b-fa41f0fc0bef",
          "locations": [{
            "Id": "us-il-ch1-azr"
          }]
        }
      ]
    }
  }
}

Obviously, you can have any number of tests, we’re showing only 2 for brevity.

Rendering a Template Test Resource

This might sound confusing but we can render the parameter variables within a general microsoft.insights/webtest model resource rather than literally writing them like above. It’s the same concept of creating a for loop with generalized integration code, just in an interesting JSON-like programming way.

In the parameters file above we have a tests array which contains each of our test descriptor objecs. A generalized version of the microsoft.insights/webtest resource we saw earlier would look like this:

{
  "name": "[parameters('tests')[copyIndex()].name]",
  "apiVersion": "2015-05-01",
  "type": "microsoft.insights/webtests",
  "location": "Central US",
  "tags": {
    "[concat('hidden-link:', resourceId('microsoft.insights/components/', parameters('appName')))]": "Resource"
  },
  "dependsOn": [
    "[concat('microsoft.insights/components/', parameters('appName'))]"
  ],
  "properties": {
    "Name": "[parameters('tests')[copyIndex()].name]",
    "Description": "[parameters('tests')[copyIndex()].description]",
    "Enabled": true,
    "Frequency": "[parameters('tests')[copyIndex()].frequency_secs]",
    "Timeout": "[parameters('tests')[copyIndex()].timeout_secs]",
    "Kind": "ping",
    "Locations": "[parameters('tests')[copyIndex(1)].locations]",
    "Configuration": {
      "WebTest": "[concat('<WebTest Name="', parameters('tests')[copyIndex(1)].name, '"',  ' Id="', parameters('tests')[copyIndex(1)].guid ,'"    Enabled="True" CssProjectStructure="" CssIteration="" Timeout="0" WorkItemIds="" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010" Description="" CredentialUserName="" CredentialPassword="" PreAuthenticate="True" Proxy="default" StopOnError="False" RecordedResultFile="" ResultsLocale="">        <Items>        <Request Method="GET" Guid="a5f10126-e4cd-570d-961c-cea43999a200" Version="1.1" Url="', parameters('tests')[copyIndex()].url ,'" ThinkTime="0" Timeout="300" ParseDependentRequests="True" FollowRedirects="True" RecordResult="True" Cache="False" ResponseTimeGoal="0" Encoding="utf-8" ExpectedHttpStatusCode="', parameters('tests')[copyIndex()].expected ,'" ExpectedResponseUrl="" ReportingName="" IgnoreHttpStatusCode="False" /></Items></WebTest>')]"
    },
    "SyntheticMonitorId": "[parameters('tests')[copyIndex()].name]"
  },
  "copy": {
    "name": "createTests",
    "count": "[length(parameters('tests'))]"
  }
}

Using the parameters template function we can grab the tests array and use the copyIndex() function which returns the current (zero-based) index of iteration. Using the length function we set the createTests iterator to the length of the array, allowing the iteration to cycle through all the elements in the tests array. Using the standard javascript array indexer syntax we can access each test object by doing parameters('tests')[copyIndex()] and appending the property such as .name or .description to access those particular fields.

Finally, using the concat function, we can render the parameterized XML by simply concatenating the replaced parameter with the necessary quotation marks and remaining xml.

One Small Caveat

As of writing this there’s a tiny bug in App Insights provisioning that doesn’t allow the modification, addition or removal of App Insights resources in parallel. Because of this we have to use the dependsOn property to force resources to provision serially. Checkout the actual template for the clever hacking to the template that needed to be done. This bug will be fixed in the coming months which will allow parallelism with App Insights.

Updating, Adding and Removing Tests

Another amazing thing about using a template like this one is that using create mode when deploying the template will update your web tests for the desired state specified by the template. So the only logic required is that which generates the template parameters file. This saves a lot of work in your deployment scripts for handling existing tests.

What this means is that existing web test resources with the same test name will just have its settings updated, without erasing the test history. When you add or remove tests from your parameters file, those tests will either be added or removed. As the template deploys, tests start running immediately after they provision, even if the entire deployment isn’t complete. This allows for maintainers to quickly know a deployment is running since tests start when they are provisioned, rather than when the entire deployment completes.

How to Dynamically Create Your Own Web Tests

Now that we’ve published a reusable template for dynamically generating web tests, you can head over to the template deployment page where you can provision web tests by generating your own parameters JSON and either using the CLI or the deploy to azure button to create the tests.

As your deployment changes, you can update existing web tests by re-deploying the template. You can also add and remove tests by simply removing or adding tests from your template parameters (just be sure to use create mode deployments).