In this post, Software Engineer Nathan Vanderby explains how to centralize your VM certificate deployment across multiple Azure Regions with ARM Templates.
Many systems still rely on certificate authentication. Those certificates need to be rotated and often managed from a central location. We know keeping certs in source control or a VM image is not only a security risk but also can be a pain for lifecycle operations.
Let’s say your topology consist of virtual machines/virtual machine scale sets in multiple regions that rely upon certificate authentication to some system. All these VMs need the same certificate. You want to manage the rotation of that certificate with as few manual steps as possible. During rotation you are required to have both the old cert and new cert on the VM before configuration/code can be updated to use the new cert. You also need to consider roll-back in case of unforeseen issues and VMs coming online before certificate rotation/configuration update is complete. Not to mention it may take hours to rollout the new certificate either due to the large number of VMs or a staged update strategy.
Azure Resource Manager (ARM) templates allow for installing certificates on VMs during deployment. A QuickStart template of this can be found here. This has a limitation where the key vault must reside in the same Azure region as the virtual machine. This is by design to avoid cross regional dependencies in the architecture.
ARM templates also allow using a key vault secret as a parameter. This does not have the same restriction of having the key vault in the same region as the resources being deployed. Additionally you can access certificates as a secret. The return value is a base64 encoding of the certificate.
In this post, the problem of pushing a certificate from a customer managed key vault to your VM is solved. This is done by copying the certificate as a secret and specially formatting it into a JSON object that is saved as a secret in a key vault in the same region as the VM, then referencing that secret during the ARM template deployment.
We can embed the script of formatting the certificate string to a JSON string into the ARM template itself using user-defined functions.
The workflow is to have the certificate from the central key vault as an input parameter to the ARM template then copy the certificate into the regional key vault (as a secret and formatted as a base64 JSON string) then push the certificate from the regional key vault to the VMs. Now if you ever need to update a certificate on a VM you just re-deploy the template.
The simplified topology could look something like this:
Note: If you’re only interested in updating the VM with the latest version of the certificate you could also accomplish this using the Azure KV VM extension. The limitation here is if a new VM is stood up (or a VM scale set scales out) during the rotation process and you need both the old and new certs on the VM this won’t download both certs.
Part 1: Copy the secret from the central Key Vault to the regional Key Vault
In order to copy the certificate across regions the certificate will be an input parameter as a secret string. Remember that certificates can be accessed the same as secrets. This will return a base64 encoding of the certificate.
Parameters File:
"parameters": { "keyVaultSecret": { "reference": { "keyVault": { "id": "/subscriptions/<SubscriptionId>/resourceGroups/<ResourceGroupName>/providers/Microsoft.KeyVault/vaults/<CentralKeyVaultName>" }, "secretName": "SampleCertificate" } } }
Template File:
"parameters": { "keyVaultSecret": { "type": "securestring" } }
Then this parameter will be added to a JSON string and set as a secret in the regional key vault that coincides with the location of the VM/VMSS. The incoming secret is in base64 and the output JSON string also needs to be in base64.
Template File:
"functions": [ { "namespace": "Sample", "members": { "convertToBase64Json": { "parameters": [ { "name": "base64Cert", "type": "string" } ], "output": { "type": "string", "value": "[base64(concat('{ \"data\":\"', parameters('base64Cert'), '\", \"dataType\": \"pfx\", \"password\": \"\" }'))]" } } } } ], "resources": [ { "type": "Microsoft.KeyVault/vaults", "name": "Region1KeyVault", ... "resources": [ { "type": "secrets", "name": "SampleCertificateAsSecret", "apiVersion": "2016-10-01", "properties": { "value": "[Sample.convertToBase64Json(parameters('keyVaultSecret'))]" }, "dependsOn": [ "Microsoft.KeyVault/vaults/Region1KeyVault" ] } ] },
Part 2: Push certificate from the regional Key Vault to the Virtual Machine
In order to push the certificate to the VM/VMSS at deployment time we simply add a secrets section to the OSProfile portion of the VM/VMSS resource.
Template File:
{ "type": "Microsoft.Compute/virtualMachines", "name": "Region1VM", ... "properties": { ... "osProfile": { "computerName": "Region1VM", ... "secrets": [ { "sourceVault": { "id": "[resourceId('Microsoft.KeyVault/vaults', Region1KeyVault)]" }, "vaultCertificates": [ { "certificateUrl": "[reference(resourceId('Microsoft.KeyVault/vaults/secrets', 'Region1KeyVault', 'SampleCertificateAsSecret')).secretUriWithVersion]", "certificateStore": "My" } ] } ], }, ... } }
Here we can see the certificate was placed in the local machine store.
Part 3: Certificate rotation
Let’s take this a step further and say you want the current version and a previous version as you rotate certificates. We can handle this by specifying a version as a part of the parameters, transforming both certificates into the properly formatted JSON as separate secrets, then pushing both secrets as certificates to the VM.
Parameters File:
"parameters": { "keyVaultSecret": { "reference": { "keyVault": { "id": "/subscriptions/<SubscriptionId>/resourceGroups/<ResourceGroupName>/providers/Microsoft.KeyVault/vaults/<CentralKeyVaultName>" }, "secretName": "SampleCertificate", "secretVersion": "cd91b2b7e10e492ebb870a6ee0591b68" } }, "keyVaultSecretSecondaryForRotation": { "reference": { "keyVault": { "id": "/subscriptions/<SubscriptionId>/resourceGroups/<ResourceGroupName>/providers/Microsoft.KeyVault/vaults/<CentralKeyVaultName>" }, "secretName": "SampleCertificate", "secretVersion": "SomeOtherVersion" } } }
For brevity I do not show the template file, but the second parameter is simply a copy of the first parameter with a different name. You will add a second secret to the key vault resource and VM/VMSS resource’s OSProfile section.
Now when it’s time to perform the cert rotation we simply update the version from one of the certificates, perform a deployment to update the VMs, then update the configs as needed to utilize the new certs, finally update the template again for both parameters to be the latest certificate version and perform another deployment to complete the rotation. This allows us at a fine grain level to control when the certificates are pushed to the VMs, allows for an easy to understand rollback process and handles new VMs spinning up during the middle of the rotation or rollback window.
References:
- About keys, secrets, and certificates –Â https://docs.microsoft.com/en-us/azure/key-vault/about-keys-secrets-and-certificates
- Azure Key Vault Virtual Machine extension now generally available –Â https://azure.microsoft.com/en-us/updates/azure-key-vault-virtual-machine-extension-now-generally-available/
- Microsoft.Compute virtualMachines template reference –Â https://docs.microsoft.com/en-us/azure/templates/microsoft.compute/2019-07-01/virtualmachines
- Microsoft.Compute virtualMachineScaleSets template reference –Â https://docs.microsoft.com/en-us/azure/templates/microsoft.compute/2019-07-01/virtualmachinescalesets
- Push a certificate onto a VM quickstart template –Â https://github.com/Azure/azure-quickstart-templates/tree/master/201-vm-push-certificate-windows
- StackOverflow “Scale set using keyvault in another region” –Â https://stackoverflow.com/questions/38856285/scale-set-using-keyvault-in-another-region
- Understand the structure and syntax of Azure Resource Manager templates –Â https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/template-syntax
- Updated: Deploy Certificates to VMs from customer-managed Key Vault –Â https://docs.microsoft.com/en-us/archive/blogs/kv/updated-deploy-certificates-to-vms-from-customer-managed-key-vault
- Use Azure Key Vault to pass secure parameter value during deployment –Â https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/key-vault-parameter
You have been very helpful. Thanks for this information. This article is really helpful for me and my team (wuschools). Thank again!