{"id":2150,"date":"2015-12-04T03:04:14","date_gmt":"2015-12-04T03:04:14","guid":{"rendered":"https:\/\/www.microsoft.com\/reallifecode\/index.php\/2015\/12\/04\/dynamic-web-test-generation-using-app-insights\/"},"modified":"2020-03-18T15:07:29","modified_gmt":"2020-03-18T22:07:29","slug":"dynamic-web-test-generation-using-app-insights","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/dynamic-web-test-generation-using-app-insights\/","title":{"rendered":"Dynamic Web Test Generation Using App Insights"},"content":{"rendered":"

SiteCore<\/a> is a business intelligence company based out of Copenhagen Denmark. They provide a solution that allows developers to understand deep business analytics in their own services. We had the opportunity to work with SiteCore\u2019s development team in Kuala Lumpur for a one-week hackfest in order to explore how they could use Application Insights<\/a>, an Azure application telemetry service, to understand the health and availability of their services.<\/p>\n

The Problem<\/h2>\n

One of the things that SiteCore does for each deployment is to run a series of recurring web tests. These are tests that ping an endpoint and expect a particular response. So for example, a test may be pseudo-coded as follows:<\/p>\n

\n
GET http:\/\/deploymenthost.com\/tests\/databaseconnection\r\n\r\nEXPECTED RESPONSE: 200\r\n<\/code><\/pre>\n<\/div>\n
The test will periodically call the specified endpoint with the specified method and headers and returns a response. When the endpoint is called by the web test the actual test logic is performed server-side. Sometimes this is ensuring a database is connected, other times it may just be checking to see if the service itself is still running. If the test fails, then an alert needs to be sent to the appropriate maintainer.<\/p>\n
SiteCore\u2019s unique problem is that they deploy a large number of services and the endpoints often change depending on new tests, host names or a combination of both. They needed a way to deploy their tests based on information known only at deploy-time.<\/p>\n
The Solution<\/h2>\n
Application Insights<\/a> offers automated web tests which can be ran from any Azure data center in the world to ensure availability for a variety of markets. You can easily create a web test from the portal:<\/p>\n
\"Web\n\"Create<\/p>\n
This works great for simple scenarios, however in the case of SiteCore, the number of tests and the properties of a test are very fluid and can change per deployment. Because of this we needed to be able to describe web tests programmatically.<\/p>\n
Applications Insights does this, but its a bit tricky at first as documentation around this is quite scarce \u2013 which hopefully this post sheds some light on.<\/p>\n
Describing a Web Test with XML<\/h2>\n
So it turns out that Web Tests are actually described by an XML schema, which coincidentally is used by Visual Studio Web Tests.<\/p>\n
We spared you the trouble of creating a webtest in Visual Studio and reverse-engineering the schema. Here\u2019s what a typical web ping test XML looks like:<\/p>\n
\n
<WebTest<\/span> Name=<\/span>\"WebTest1\"<\/span> Id=<\/span>\"0000-0000-0000-0000\"<\/span> Enabled=<\/span>\"True\"<\/span> CssProjectStructure=<\/span>\"\"<\/span> CssIteration=<\/span>\"\"<\/span> Timeout=<\/span>\"0\"<\/span> WorkItemIds=<\/span>\"\"<\/span> xmlns=<\/span>\"http:\/\/microsoft.com\/schemas\/VisualStudio\/TeamTest\/2010\"<\/span> Description=<\/span>\"\"<\/span> CredentialUserName=<\/span>\"\"<\/span> CredentialPassword=<\/span>\"\"<\/span> PreAuthenticate=<\/span>\"True\"<\/span> Proxy=<\/span>\"default\"<\/span> StopOnError=<\/span>\"False\"<\/span> RecordedResultFile=<\/span>\"\"<\/span> ResultsLocale=<\/span>\"\"<\/span>><\/span>\r\n  <Items><\/span>\r\n    <Request<\/span> Method=<\/span>\"GET\"<\/span> Guid=<\/span>\"a5f10126-e4cd-570d-961c-cea43999a200\"<\/span> Version=<\/span>\"1.1\"<\/span> Url=<\/span>\"http:\/\/microsoft.com\"<\/span> ThinkTime=<\/span>\"0\"<\/span> Timeout=<\/span>\"300\"<\/span> ParseDependentRequests=<\/span>\"True\"<\/span> FollowRedirects=<\/span>\"True\"<\/span> RecordResult=<\/span>\"True\"<\/span> Cache=<\/span>\"False\"<\/span> ResponseTimeGoal=<\/span>\"0\"<\/span> Encoding=<\/span>\"utf-8\"<\/span> ExpectedHttpStatusCode=<\/span>\"200\"<\/span> ExpectedResponseUrl=<\/span>\"\"<\/span> ReportingName=<\/span>\"\"<\/span> IgnoreHttpStatusCode=<\/span>\"False\"<\/span> \/><\/span>\r\n  <\/Items><\/span>\r\n<\/WebTest><\/span>\r\n<\/code><\/pre>\n<\/div>\n
It turns out that in Azure Resource Manager we can deploy App Insights web test resources which are of the resource type Microsoft.Insights\/webtests<\/code>.<\/p>\n
A Microsoft.Insights\/webtests<\/code> resource may look like this (a referenced Microsoft.Insights\/component<\/code> is omitted for brevity):<\/p>\n
\n
{<\/span>\r\n  \"name\"<\/span>:<\/span> \"Test1\"<\/span>,<\/span>\r\n  \"apiVersion\"<\/span>:<\/span> \"2015-05-01\"<\/span>,<\/span>\r\n  \"type\"<\/span>:<\/span> \"microsoft.insights\/webtests\"<\/span>,<\/span>\r\n  \"location\"<\/span>:<\/span> \"Central US\"<\/span>,<\/span>\r\n  \"tags\"<\/span>:<\/span> {<\/span>\r\n    \"[concat('hidden-link:', resourceId('microsoft.insights\/components\/appinsightsresource')))]\"<\/span>:<\/span> \"Resource\"<\/span>\r\n  },<\/span>\r\n  \"dependsOn\"<\/span>:<\/span> [<\/span>\r\n    \"[concat('microsoft.insights\/components\/', parameters('appName'))]\"<\/span>\r\n  ],<\/span>\r\n  \"properties\"<\/span>:<\/span> {<\/span>\r\n    \"Name\"<\/span>:<\/span> \"Test1\"<\/span>,<\/span>\r\n    \"Description\"<\/span>:<\/span> \"A healthy descriptive piece of text\"<\/span>,<\/span>\r\n    \"Enabled\"<\/span>:<\/span> true<\/span>,<\/span>\r\n    \"Frequency\"<\/span>:<\/span> 300<\/span>,<\/span>\r\n    \"Timeout\"<\/span>:<\/span> 30<\/span>,<\/span>\r\n    \"Kind\"<\/span>:<\/span> \"ping\"<\/span>,<\/span>\r\n    \"Locations\"<\/span>:<\/span> [{<\/span> \"Id\"<\/span>:<\/span> \"us-il-ch1-azr\"<\/span> }],<\/span>\r\n    \"Configuration\"<\/span>:<\/span> {<\/span>\r\n      \"WebTest\"<\/span>:<\/span> \"<WebTest Name=\"<\/span>WebTest<\/span>1<\/span>\" Id=\"<\/span>0000-0000-0000-0000<\/span>\" Enabled=\"<\/span>True<\/span>\" CssProjectStructure=\"\" CssIteration=\"\" Timeout=\"<\/span>0<\/span>\" WorkItemIds=\"\" xmlns=\"<\/span>http:\/\/microsoft.com\/schemas\/VisualStudio\/TeamTest\/<\/span>2010<\/span>\" Description=\"\" CredentialUserName=\"\" CredentialPassword=\"\" PreAuthenticate=\"<\/span>True<\/span>\" Proxy=\"<\/span>default<\/span>\" StopOnError=\"<\/span>False<\/span>\" RecordedResultFile=\"\" ResultsLocale=\"\"><Items><Request Method=\"<\/span>GET<\/span>\" Guid=\"<\/span>a<\/span>5<\/span>f<\/span>10126<\/span>-e<\/span>4<\/span>cd<\/span>-570<\/span>d<\/span>-961<\/span>c-cea<\/span>43999<\/span>a<\/span>200<\/span>\" Version=\"<\/span>1.1<\/span>\" Url=\"<\/span>http:\/\/microsoft.com<\/span>\" ThinkTime=\"<\/span>0<\/span>\" Timeout=\"<\/span>300<\/span>\" ParseDependentRequests=\"<\/span>True<\/span>\" FollowRedirects=\"<\/span>True<\/span>\" RecordResult=\"<\/span>True<\/span>\" Cache=\"<\/span>False<\/span>\" ResponseTimeGoal=\"<\/span>0<\/span>\" Encoding=\"<\/span>utf<\/span>-8<\/span>\" ExpectedHttpStatusCode=\"<\/span>200<\/span>\" ExpectedResponseUrl=\"\" ReportingName=\"\" IgnoreHttpStatusCode=\"<\/span>False<\/span>\" \/><\/Items><\/WebTest>\"<\/span>\r\n    },<\/span>\r\n    \"SyntheticMonitorId\"<\/span>:<\/span> \"Test1\"<\/span>\r\n  }<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n<\/div>\n
After discovering this, you can tell that it would be easy to programmatically create web tests by using a template XML file and rendering that XML file with test-specific parameters such as the expected HTTP status code or the URL for the test to use.<\/p>\n
SiteCore originally wanted to have this logic within a Powershell script they currently use to create their deployments. They would have to use the Azure Resource Manager APIs to create the resources individually and manage the deployment of each. This works, but brings additional complexity and management to the deployment logic.<\/p>\n
It turns out there is an easier way to do this. Using Azure Resource Manager, we can leverage desired-state deployments with a bit of clever template writing.<\/p>\n
Writing Advanced Azure Resource Management Templates<\/h2>\n
You may have heard of Azure Resource Management templates which allow you to describe infrastructure by extending JSON syntax. What you may have not known is that Azure Resource Manager Template Language<\/a> comes with a variety of features to bring some basic logic such as iteration and arithmetic into a template allowing for more complex deployments with much less code.<\/p>\n
Resource Manager templates have the ability to deploy resources iteratively using the copy<\/code> resource management property. For a given resource, using the copy<\/code> property you can specify the name of the copy operation as well the number of times it should iterate:<\/p>\n
\n
\"copy\"<\/span>:<\/span> {<\/span>\r\n  \"name\"<\/span>:<\/span> \"createTests\"<\/span>,<\/span>\r\n  \"count\"<\/span>:<\/span> 10<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n<\/div>\n
Using a parameters file which passes an array of objects, we can use each object to describe all the necessary information for each test:<\/p>\n
\n
{<\/span>\r\n  \"$schema\"<\/span>:<\/span> \"http:\/\/schema.management.azure.com\/schemas\/2015-01-01\/deploymentParameters.json#\"<\/span>,<\/span>\r\n  \"contentVersion\"<\/span>:<\/span> \"1.0.0.0\"<\/span>,<\/span>\r\n  \"parameters\"<\/span>:<\/span> {<\/span>\r\n    \"appName\"<\/span>:<\/span> {<\/span>\r\n      \"value\"<\/span>:<\/span> \"someapp\"<\/span>\r\n    },<\/span>\r\n    \"emails\"<\/span>:<\/span> {<\/span>\r\n      \"value\"<\/span>:<\/span> [<\/span>\r\n        \"youremail@yourdomain.com\"<\/span>\r\n      ]<\/span>\r\n    },<\/span>\r\n    \"tests\"<\/span>:<\/span> {<\/span>\r\n      \"value\"<\/span>:<\/span> [<\/span>\r\n        {<\/span>\r\n          \"name\"<\/span>:<\/span> \"test1\"<\/span>,<\/span>\r\n          \"url\"<\/span>:<\/span> \"http:\/\/www.microsoft.com\"<\/span>,<\/span>\r\n          \"expected\"<\/span>:<\/span> 200<\/span>,<\/span>\r\n          \"frequency_secs\"<\/span>:<\/span> 300<\/span>,<\/span>\r\n          \"timeout_secs\"<\/span>:<\/span> 30<\/span>,<\/span>\r\n          \"failedLocationCount\"<\/span>:<\/span> 1<\/span>,<\/span>\r\n          \"description\"<\/span>:<\/span> \"a description for test1\"<\/span>,<\/span>\r\n          \"guid\"<\/span>:<\/span> \"cc1c4b95-0a39-48ce-9c7b-fa41f0fc0bee\"<\/span>,<\/span>\r\n          \"locations\"<\/span>:<\/span> [{<\/span>\r\n            \"Id\"<\/span>:<\/span> \"us-il-ch1-azr\"<\/span>\r\n          }]<\/span>\r\n        },<\/span>\r\n        {<\/span>\r\n          \"name\"<\/span>:<\/span> \"test2\"<\/span>,<\/span>\r\n          \"url\"<\/span>:<\/span> \"http:\/\/www.microsoft.com\"<\/span>,<\/span>\r\n          \"expected\"<\/span>:<\/span> 404<\/span>,<\/span>\r\n          \"frequency_secs\"<\/span>:<\/span> 300<\/span>,<\/span>\r\n          \"timeout_secs\"<\/span>:<\/span> 30<\/span>,<\/span>\r\n          \"failedLocationCount\"<\/span>:<\/span> 1<\/span>,<\/span>\r\n          \"description\"<\/span>:<\/span> \"a description for test3\"<\/span>,<\/span>\r\n          \"guid\"<\/span>:<\/span> \"cc1c4b95-0a39-48ce-9c7b-fa41f0fc0bef\"<\/span>,<\/span>\r\n          \"locations\"<\/span>:<\/span> [{<\/span>\r\n            \"Id\"<\/span>:<\/span> \"us-il-ch1-azr\"<\/span>\r\n          }]<\/span>\r\n        }<\/span>\r\n      ]<\/span>\r\n    }<\/span>\r\n  }<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n<\/div>\n
Obviously, you can have any number of tests, we\u2019re showing only 2 for brevity.<\/p>\n
Rendering a Template Test Resource<\/h3>\n
This might sound confusing but we can render the parameter variables within a general<\/em> microsoft.insights\/webtest<\/code> model resource rather than literally writing them like above. It\u2019s the same concept of creating a for<\/code> loop with generalized integration code, just in an interesting JSON-like programming way.<\/p>\n
In the parameters file above we have a tests<\/code> array which contains each of our test descriptor objecs. A generalized<\/em> version of the microsoft.insights\/webtest<\/code> resource we saw earlier would look like this:<\/p>\n
\n
{<\/span>\r\n  \"name\"<\/span>:<\/span> \"[parameters('tests')[copyIndex()].name]\"<\/span>,<\/span>\r\n  \"apiVersion\"<\/span>:<\/span> \"2015-05-01\"<\/span>,<\/span>\r\n  \"type\"<\/span>:<\/span> \"microsoft.insights\/webtests\"<\/span>,<\/span>\r\n  \"location\"<\/span>:<\/span> \"Central US\"<\/span>,<\/span>\r\n  \"tags\"<\/span>:<\/span> {<\/span>\r\n    \"[concat('hidden-link:', resourceId('microsoft.insights\/components\/', parameters('appName')))]\"<\/span>:<\/span> \"Resource\"<\/span>\r\n  },<\/span>\r\n  \"dependsOn\"<\/span>:<\/span> [<\/span>\r\n    \"[concat('microsoft.insights\/components\/', parameters('appName'))]\"<\/span>\r\n  ],<\/span>\r\n  \"properties\"<\/span>:<\/span> {<\/span>\r\n    \"Name\"<\/span>:<\/span> \"[parameters('tests')[copyIndex()].name]\"<\/span>,<\/span>\r\n    \"Description\"<\/span>:<\/span> \"[parameters('tests')[copyIndex()].description]\"<\/span>,<\/span>\r\n    \"Enabled\"<\/span>:<\/span> true<\/span>,<\/span>\r\n    \"Frequency\"<\/span>:<\/span> \"[parameters('tests')[copyIndex()].frequency_secs]\"<\/span>,<\/span>\r\n    \"Timeout\"<\/span>:<\/span> \"[parameters('tests')[copyIndex()].timeout_secs]\"<\/span>,<\/span>\r\n    \"Kind\"<\/span>:<\/span> \"ping\"<\/span>,<\/span>\r\n    \"Locations\"<\/span>:<\/span> \"[parameters('tests')[copyIndex(1)].locations]\"<\/span>,<\/span>\r\n    \"Configuration\"<\/span>:<\/span> {<\/span>\r\n      \"WebTest\"<\/span>:<\/span> \"[concat('<WebTest Name=\"', parameters('tests')[copyIndex(1)].name, '\"',  ' Id=\"', parameters('tests')[copyIndex(1)].guid ,'\"    Enabled=\"True\" CssProjectStructure=\"\" CssIteration=\"\" Timeout=\"0\" WorkItemIds=\"\" xmlns=\"http:\/\/microsoft.com\/schemas\/VisualStudio\/TeamTest\/2010\" Description=\"\" CredentialUserName=\"\" CredentialPassword=\"\" PreAuthenticate=\"True\" Proxy=\"default\" StopOnError=\"False\" RecordedResultFile=\"\" ResultsLocale=\"\">        <Items>        <Request Method=\"GET\" Guid=\"a5f10126-e4cd-570d-961c-cea43999a200\" Version=\"1.1\" Url=\"', parameters('tests')[copyIndex()].url ,'\" ThinkTime=\"0\" Timeout=\"300\" ParseDependentRequests=\"True\" FollowRedirects=\"True\" RecordResult=\"True\" Cache=\"False\" ResponseTimeGoal=\"0\" Encoding=\"utf-8\" ExpectedHttpStatusCode=\"', parameters('tests')[copyIndex()].expected ,'\" ExpectedResponseUrl=\"\" ReportingName=\"\" IgnoreHttpStatusCode=\"False\" \/><\/Items><\/WebTest>')]\"<\/span>\r\n    },<\/span>\r\n    \"SyntheticMonitorId\"<\/span>:<\/span> \"[parameters('tests')[copyIndex()].name]\"<\/span>\r\n  },<\/span>\r\n  \"copy\"<\/span>:<\/span> {<\/span>\r\n    \"name\"<\/span>:<\/span> \"createTests\"<\/span>,<\/span>\r\n    \"count\"<\/span>:<\/span> \"[length(parameters('tests'))]\"<\/span>\r\n  }<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n<\/div>\n
Finally, using the concat<\/code> function<\/a>, we can render the parameterized XML by simply concatenating the replaced parameter with the necessary quotation marks and remaining xml.<\/p>\n
One Small Caveat<\/h3>\n
As of writing this there\u2019s a tiny bug in App Insights provisioning that doesn\u2019t allow the modification, addition or removal of App Insights resources in parallel. Because of this we have to use the dependsOn<\/code> property to force resources to provision serially. Checkout the actual template<\/a> for the clever hacking to the template that needed to be done. This bug will be fixed in the coming months which will allow parallelism with App Insights.<\/p>\n
Updating, Adding and Removing Tests<\/h3>\n
Another amazing thing about using a template like this one is that using create<\/strong> mode when deploying the template will update your web tests for the desired state specified by the template. So the only logic required is that which generates the template parameters file. This saves a lot of work in your deployment scripts for handling existing tests.<\/p>\n
What this means is that existing web test resources with the same test name will just have its settings updated, without erasing the test history. When you add or remove tests from your parameters file, those tests will either be added or removed. As the template deploys, tests start running immediately after they provision, even if the entire deployment isn\u2019t complete. This allows for maintainers to quickly know a deployment is running since tests start when they are provisioned, rather than when the entire deployment completes.<\/p>\n
How to Dynamically Create Your Own Web Tests<\/h2>\n
Now that we\u2019ve published a reusable template<\/a> for dynamically generating web tests, you can head over to the template deployment page<\/a> where you can provision web tests by generating your own parameters JSON and either using the CLI<\/a> or the deploy to azure<\/strong> button to create the tests.<\/p>\n
As your deployment changes, you can update existing web tests by re-deploying the template. You can also add and remove tests by simply removing or adding tests from your template parameters (just be sure to use create<\/strong> mode deployments).<\/p>\n","protected":false},"excerpt":{"rendered":"
Learn how leverage complex Azure Resource Manager Template Language to dynamically create and update Web availability Tests.<\/p>\n","protected":false},"author":21345,"featured_media":11151,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[10],"tags":[60,64,328,349],"class_list":["post-2150","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure-app-services","tag-azure","tag-azure-application-insights","tag-sitecore","tag-telemetry"],"acf":[],"blog_post_summary":"
Learn how leverage complex Azure Resource Manager Template Language to dynamically create and update Web availability Tests.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/2150","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/21345"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=2150"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/2150\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media\/11151"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=2150"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=2150"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=2150"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Using the parameters<\/code> template function<\/a> we can grab the tests<\/code> array and use the copyIndex()<\/code> function<\/a> which returns the current (zero-based) index of iteration. Using the length function<\/a> we set the createTests<\/code> iterator to the length of the array, allowing the iteration to cycle through all the elements in the tests<\/code> array. Using the standard javascript array indexer syntax we can access each test object by doing parameters('tests')[copyIndex()]<\/code> and appending the property such as .name<\/code> or .description<\/code> to access those particular fields.<\/p>\n
The next question you may ask is, \u2018How does the template know how many and what kind of tests to deploy\u2019. Using a template parameters file, we can actually pass more complex template parameters than what you may see on a typical Azure Resource Manager template parameters file<\/a>.<\/p>\n