Using VSTS Service Hooks in Release Management to add resource cleanup resilience
In this post, Principal Consultant/App Dev Manager Lou Sawyer describes how to integrate webhooks and runbooks to cleanup failed Azure deployments.
Release Management capabilities in Visual Studio Team Service are fantastic and offers extensibility points to make everyone’s life easier. In this post, we’ll be discussing the Service Hooks feature along with a potential use case integrating with a Runbook.
As detailed in Integrate with service hooks, Service Hooks allow you to perform tasks in other services in response to events in VSTS. This capability can be used to start a runbook from a webhook (runbooks and use cases are covered in Azure Automation User Documentation) to perform conditional tasks after a VSTS Release.
One common use of Release Deployments is to perform testing that can involve a significant number of resources in Azure to complete. By default, when a deployment fails in VSTS, it stops the deployment at that time and does not continue in order to give you a chance to investigate the failure. This can lead to resources continuing to be available longer than needed adding cost to the environment. The option does exist to configure to continue on error, however this is not conditional and may result in more task failure (or the environment being deleted when you would want to investigate). In such a scenario, use a webhook to starts a runbook on deployment failure to examine which task failed and clean-up the environment.
Consider the image shown below, in this case there are intermittent failures and this release creates resources in a resource group in Azure. Depending on when it fails, these resources may result in unwanted overhead.
To mitigate this scenario, consider using a runbook. The image below shows a PowerShell Workflow runbook being created to handle a failed release event
The workflow needs to take one parameter named webhookData which will contain the body of the post request from VSTS. The request body of webhookData then needs to be deserialized so we can examine the release details. In the code below, we are walking through the environment deployment phase tasks to find the first task where we know the resourceGroupName and location are used as parameters. We can use these values to test if the resourceGroupName exists at the specified location and remove it; a more elaborate script would iterate through all the tasks to find where it failed and then decide to remove the resource group or not.
The next step is to create a webhook for the runbook. This is covered fairly well in Starting an Azure Automation runbook with a webhook. As a word of caution, the URL for the webhook cannot be retrieved after creation, so save it before you click Create!
At this point, we need to create the Service Hook in VSTS. This can be done from a Service Hooks settings as shown below
From the Service Hooks pane, you have the option to create a subscription, choose it to create a Service hook.
There are lots of services you can send events to, in this case we are interested in Web hooks; this topic is covered in VSTS documentation on Web Hooks.
The trigger options can be narrowed to a specific release, environment, and status. In this case, we are interested in failed deployments to UAT for TailSpin Toys as shown below.
The next window allows you to paste the URI generated when you created the webhook for your runbook automation. There are a lot of options for message detail and it’s important to note that the highest level of detail (All) is the default. This option contains a lot of information about your release so it’s important to use secure connections. In this case, we need the full release details to pull out specific information from the tasks and any conditional work we want to do.
Finally, the service hook is done and the runbook is ready. The next time a deployment fails, you should see an output in your runbook jobs like the screen below. As the snapshot shows, the failed deployment resource groups were found and removed.
There is a lot of power and flexibility in the Service Hooks feature in VSTS. Hopefully this gives some additional ideas on how to do conditional processing on a release that needs attention.
Premier Support for Developers provides strategic technology guidance, critical support coverage, and a range of essential services to help teams optimize development lifecycles and improve software quality. Contact your Application Development Manager (ADM) or email us to learn more about what we can do for you.
Light
Dark
0 comments