Public preview of Workload identity federation for Azure Pipelines

Eric van Wijk

September 11th, 202335 11

Do you want to stop storing secrets and certificates in Azure service connections? Are you tired rotating these secrets whenever they expire? We are now announcing a public preview of workload identity federation for Azure service connections. Workload identity federation uses an industry-standard technology, Open ID Connect (OIDC), to simplify the authentication between Azure Pipelines and Azure. Instead of secrets, a federation subject is used to facilitate this authentication.

How it works

As part of this feature, the Azure (ARM) service connection has been updated with an additional scheme to support workload identity federation. This allows Pipeline tasks that use the Azure service connection to authenticate using a federation subject (sc://<org>/<project>/<service connection name>). The main benefits of using this scheme over existing authentication schemes are as follows:

  • Simplified management: You do not need to generate, copy, and store secrets from service principals in Azure Entra ID to Azure DevOps anymore. Secrets that are used in other authentication schemes of Azure service connections (e.g., service principal) expire after a certain period (2 years currently). When they expire, pipelines fail. You have to generate a new secret and update the service connection. Switching to workload identity federation eliminates the need to manage these secrets and improves the overall experience of creating and managing service connections.
  • Improved security: With workload identity federation, the federation subject (sc://<org>/<project>/<service connection name>) uniquely identifies what the identity can be used for, which provides a better constraint than a (shared) secret. There is no persistent secret involved in the communication between Azure Pipelines and Azure. As a result, tasks running in pipeline jobs cannot leak or exfiltrate secrets that have access to your production environments. This has often been a concern for our customers.

How can I use it?

You can take advantage of workload identity federation in two ways:

  • Convert your existing Azure service connections based on secrets to the new scheme. You can perform this conversion one connection at a time. Best of all, you do not have to modify any of the pipelines that use those service connections. They will automatically leverage the new scheme once you complete the conversion.
  • Use the new workload identity federation scheme whenever you create a new Azure service connection. Moving forward, this will be the recommended method.

Converting an existing Azure service connection

To convert a previously created Azure service connection, select the “Convert” action after selecting the connection:

You can roll back from a conversion to use a secret by clicking the revert link on the service connection details page:

Create a new Azure service connection

To create a new Azure service connection using workload identity federation, simply select Workload identity federation (automatic) in the Azure service connection creation experience:

Create manually with a Managed Identity or Service Principal

You can also create an Azure service connection using workload identity federation manually with either a Service Principal or Managed Identity. Developers that do not have permission to create App Registrations in Azure Entra ID (AAD) can create Azure service connections with Workload identity federation by configuring the federation on a Managed Identity instead of a Service Principal. Using a Managed Identity in federation does not require assignment and works across all agent hosting models (Hosted, Self-hosted, Scale set agents). See manual configuration instructions for a step by step guide.

Built-in Pipeline tasks

Azure Pipelines built-in tasks that target Azure have been updated to take advantage of workload identity federation. This is the complete list: AzureAppServiceManage, AzureAppServiceSettings, AzureCLI, AzureCloudPowerShellDeployment, AzureContainerApps, AzureFunctionAppContainer, AzureFunctionApp, AzureKeyVault, AzureMonitor, AzureMysqlDeployment, AzurePolicy, AzurePowerShell, AzureResourceGroupDeployment, AzureResourceManagerTemplateDeployment, AzureRmWebAppDeployment, AzureSpringCloud, AzureVmssDeployment, AzureWebAppContainer, AzureWebApp, DockerCompose, Docker, HelmDeploy, InvokeRestApi, JavaToolInstaller, JenkinsDownloadArtifacts, Kubernetes.

Azure CLI task support for inline authentication

If a tool does not have a task or you prefer to use tools directly inside a script you provide, you can use the AzureCLI@2 task task to access the federated token used during authentication. Setting property addSpnToEnvironment: true will populate the idToken, servicePrincipalId and tenantId environment variables.

This example shows inline configuration of Terraform azuread/azurerm providers:

  - task: AzureCLI@2
    inputs:
      addSpnToEnvironment: true
      azureSubscription: 'my-azure-service-connection'
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        # Inherit Azure CLI service connection
        export ARM_CLIENT_ID=$servicePrincipalId
        export ARM_OIDC_TOKEN=$idToken
        export ARM_TENANT_ID=$tenantId
        export ARM_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
        export ARM_USE_OIDC=true

        terraform init
        terraform apply -auto-approve

The idToken has a lifetime of 10 minutes.

Custom Azure tasks and extensions

If you are using a task from the Marketplace or a home-grown custom task to deploy to Azure, then it may not support workload identity federation yet. In these cases, we ask task developers to support workload identity federation to improve security.

Image oidc collaboration

Tasks that take a connectedService:AzureRM input in task.json can be updated so support workload identity federation with the following steps:

  • Request an idToken using the new Oidctoken REST API (arrow 1 in above diagram).
  • Exchange the idToken for an access token using the federated credential flow of the OAuth API, specifying the idToken as client_assertion (arrows 2 & 4 in above diagram);
    or:
  • For tasks that act as a wrapper around a tool that performs authentication itself, use the tools’ authentication method to specify the federated token.

You can also publish extensions to the Marketplace with Workload identity federation. This blog post explains how.

Terraform tasks

The Terraform DevLabs and Azure Pipelines Terraform Tasks extensions have been updated to support workload identity federation. The Terraform DevLabs extension uses the azure-pipelines-tasks-artifacts-common npm package to get the idToken, see code example.

If you are a Terraform user, you should review this great walkthrough on the Tech Community that uses the Azure-Samples/azure-devops-terraform-oidc-ci-cd repository to create and use Azure service connections with Terraform.

Public preview

The feature will roll out over the next few weeks. For this preview, we support workload identity federation only for Azure service connections. This scheme does not work with any other types of service connections. See our docs for more details.

Eric van Wijk Product Manager, Azure DevOps

Read next

35 comments

Discussion is closed. Login to edit/delete existing comments.

  • David Corrigan 2

    Where is the documentation for creating service connections to non-Azure services? Can additional claims and audiences be configured either in a service connection credential configuration or when requesting the OIDC token in the pipeline?

    • David Corrigan 0

      Guess the preview is officially only for Azure connections, so the question is when will it be available for custom connections? We prototyped some other connection types (Hashicorp Vault, JFrog Artifactory, k8s) and it mostly worked great, but waiting on the official availability to schedule our work to get the final versions written and deployed so we can stop relying on hard coded credentials for those services.

      • Eric van WijkMicrosoft employee 0

        Support for custom Service Connections will indeed come later.

        • Lovett, Alex 0

          is this road-mapped? We want to allow ADO Services to control resources in our AWS tenancy. Is this possible now without creds?

          • anonymous 0

            this comment has been deleted.

  • James Dawson 0

    Nice to see this feature arrive for Azure DevOps, we’ve been using it in GitHub Actions for a little while. Is the 10 minute lifetime of the idToken ever likely to be configurable?

    This has proven to been quite a showstopper for us in GitHub, where we have scripted deployment processes that run longer than 10 minutes.

    Sometimes this is fine, as the Azure CLI or Azure PowerShell access tokens have a longer lifetime, however, when a deployment process includes a step towards the end that involves requesting an access token for another service (e.g. a data plane operation), then this fails as the idToken has expired. For example, trying to make configuration changes to a Synapse Workspace once ARM has provisioned it.

    • Eric van WijkMicrosoft employee 0

      I don’t expect idToken lifetime to be an issue. An idToken is acquired for every task run that consumes a Service Connection that uses Workload identity federation. As long as an idToken is exchanged for an AAD token at the start of a task run (which is the case for all Built-in tasks), it is the AAD token lifetime that matters. Let me know if you have an actual issue.

      • James Dawson 1

        Interesting – thanks for the reply. I’ll convert a GHA workflow we setup to repro the issue and see what happens.

        UPDATE: Having now done the above, I can confirm that we don’t see the equivalent token expiry issue when running in Azure Pipelines. Many Thanks!

      • Brett Hoggins 0

        Hi Eric, seems we may have come across an issue for this when running the new “terraform test” framework. Because terraform runs each test sequentially, it seems to acquire a new AAD token at the start of each test run. For long-running tests they eventually start failing as the original IdToken has expired.

        • Eric van WijkMicrosoft employee 0

          Hi Brett, thanks for letting us know. You can request a new idToken with the Oidctoken REST API and assign it to the ARM_OIDC_TOKEN environment variable before each terraform (test) statement. Would that work in this case?

          • Brett Hoggins 1

            Hi Eric, thanks for the tip, but it looks like getting the token via that method has the same timeout.

            Terraform searches for and runs all tests found in the ./tests folder sequentially, so the only workaround I can think of would be to search for each test definition file, and run each one at a time using the filter parameter – fetching a new idToken before each test. But even this isn’t guaranteed as you can have multiple tests per definition file.

            Customizable timeout would be ideal…

            For the record, using the REST api turned out to be quite a headache! Here’s the code if anyone would like to try it:

                  #serviceConnectionId=$(az devops invoke \
      #  --area serviceendpoint \
      #  --resource endpoints \
      #  --route-parameters \
      #      project=$(System.TeamProjectId) \
      #  --query-parameters  \
      #      endpointNames='$(serviceConnection)' \
      #  -o json | jq -r .value[0].id
      #)
      ## Seems the above does not work due to the system access token not having the correct permissions
      ## Using hard-coded value instead :(
      serviceConnectionId="000000-0000-0000-0000-000000000000"

      customToken=$(az devops invoke \
        --http-method POST \
        --area distributedtask \
        --resource oidctoken \
        --organization $(System.CollectionUri) \
        --route-parameters \
            scopeIdentifier=$(System.TeamProjectId) \
            hubName=$(System.HostType) \
            planId=$(System.PlanId) \
            jobId=$(System.JobId) \
        --query-parameters \
            serviceConnectionId=$serviceConnectionId \
        --api-version=7.2-preview \
        -o json | jq -r .oidcToken
      )

      ARM_CLIENT_ID=$servicePrincipalId
      #ARM_OIDC_TOKEN=$idToken
      ARM_OIDC_TOKEN=$customToken
      ARM_TENANT_ID=$tenantId
      ARM_SUBSCRIPTION_ID=$(az account show --query id -o tsv)
      ARM_USE_OIDC=true
        • Eric van WijkMicrosoft employee 0

          Hi Brent. Here’s a script to do the same. Can you create an issue in the Developer Community and tag me? We may need to work with Hashicorp to address this, as I expect this to be a broader issue than Azure Pipelines.

        • Eric van WijkMicrosoft employee 0

          Hi Brett,

          You can tell Terraform to execute a single test file with terraform test -verbose -filter=. This provides the opportunity to acquire a new idToken before each test file execution. See examples in bash & PowerShell.

        • Jared HolgateMicrosoft employee 1

          For anyone else seeing this issue, there is a solution here. Thanks

  • Bill Kan 0

    I am using docker built in task for service connection and it’s not asking me to convert to workload identity federation. I do see it in my other arm connections just not docker.

    • Eric van WijkMicrosoft employee 0

      Correct, we will add support to use Workload identity federation for Azure Container Registry to Docker Registry Service Connections later.

  • Shobit Mahajan 0

    I would love to see this work for other cloud services as well. I used OIDC to connect to my AWS account using Gitlab. I am looking forward to connect to my AWS account from Azure Pipelines as well. I hope support for this is coming soon. 🙂

  • Daniel Schroeder 0

    This is great. I’ve used it to convert some of our existing Azure Resource Manager service connections that were using the Service Principal Authentication. Oddly though the “Convert” button does not show up on all of the Azure Resource Manager service connections. Even within the same team project, some service connections have the Convert button, while others don’t. As far as I can tell there is nothing different about the service connections besides the Azure resource that they connect to. The

    1. Do you know why the Convert button only shows up on some service connections, but not others?
    2. Is there a way to manually convert a service connection to use Workload Identity Federation when the Convert button is not an option?
    • Eric van WijkMicrosoft employee 0

      Hi Daniel. Convert works only for App registrations that can be modified by the user editing the Service Connection, typically the creator of the Service Connection.

  • Berkvens, Allan 0

    This does not work with task AzureFileCopy@5. Is there a timeline on when this will be available, or a workaround we can use in the mean time?

    • Eric van WijkMicrosoft employee 1

      We will update AzureFileCopy@5 as soon as AzCopy supports Workload identity federation. This issue tracks that work.

      UPDATE: The AzureFileCopy@6 task supports Workload identity federation.

  • Ihar Statkevich 0

    Hi. I’ve got some issue regarding this oidc token. When I use variable group linked to Key Vault and try to re-run failed job I see this error:

    [error]Unhandled: Cannot read properties of null (reading ‘oidcToken’)

    [error]TypeError: Cannot read properties of null (reading ‘oidcToken’)

    at /home/vsts/work/_tasks/AzureKeyVault_1e244d32-2dd4-4165-96fb-b7441ca9331e/1.228.0/node_modules/azure-pipelines-tasks-azure-arm-rest/azure-arm-common.js:157:35
at processTicksAndRejections (node:internal/process/task_queues:96:5)
    • Eric van WijkMicrosoft employee 0

      Hi Ihar. In what type of job is this happening? e.g. agentless, deployment group, etc.

  • Mark Grills 0

    Hi Eric,

    Regarding the token expiry and the statement of the idToken having a lifetime of 10 minutes. Is the expiry period configurable or is it a fixed expiry period?

    Alternatively, is the expiry period of the token used by the Azure Devops tasks greater than 10 minutes?

    Our Azure DevOps pipelines execute terraform and some deployment of Azure resources can be in excess of an hour. Wondering if this is suitable for such deployments.

    Thanks
    Mark

    • Eric van WijkMicrosoft employee 1

      Hi Mark, the idToken is valid for 10 minutes. Azure Pipelines tasks exchange the idToken for an AAD bearer token at the start of executing the task. In the case of Terraform, the tool the task is encapsulating will do so. Running Terraform apply for longer than 10 minutes is not an issue as the AAD bearer token has a longer lifetime.

      Thanks, Eric

  • Anil Choubey 0

    Above its mentioned that “Developers that do not have permission to create App Registrations in Azure Entra ID (AAD) can create Azure service connections with Workload identity federation by configuring the federation on a Managed Identity” . This requires some guardrail in production environent to control who can create a managed identity , is there any guardrail available on Azure to control this or a clarity of what roles and permissions is needed for creating a managed identity on Azure.

    • Eric van WijkMicrosoft employee 0

      Hi Anil, this doc describes how to configure Managed Identity for federation. You will have to create the Managed Identity yourself, we will not create it for you. You can create an Azure policy that governs the creation of federations on Managed Identity, see doc.

  • Liza RudakovaMicrosoft employee 0

    This is a great feature. Can it be used to deploy to Service Fabric clusters, or are they not yet supported?

Feedback usabilla icon