Updates to Approvals and Checks

Silviu

June 11th, 202311 6

Approvals and Checks provide increased security to your YAML pipelines. They allow you to control if a pipeline run is allowed to access a resource.

Let’s look at an example. Say you develop the FabrikamFiber web app. To deploy a new version of your app, you use a YAML pipeline that uses an Azure Resource Manager (ARM) service connection. Your deployment to production policy requires the app meet some performance criteria. You implemented the policy using an Invoke Azure Function check on the ARM service connection. Your Azure Function can check the performance of the to-be-deployed version of the system against a test environment and decide accordingly.

We are making changes to improve the scalability of Invoke Azure Function & REST API checks and customer experience.

Scalability improvements

We noticed that when an organization makes extensive use of the Invoke Azure Function & REST API checks, they do not scale. That is, a modest increase in the number of running checks leads to abnormally large delays in checks execution time, negatively impacting deployment experience.

Our solution to scale Invoke Azure Function and REST API checks is to enforce their configuration to match recommended usage. If you followed our guidelines, your checks are compliant and need no further changes.

Invoke Azure Function & REST API Checks can run in two modes:

  • ApiResponse mode (or Synchronous mode): In this mode, Azure DevOps makes a REST HTTP call to your check and actively waits for an answer.
  • Callback mode (or Asynchronous mode): In this mode, Azure DevOps still makes a REST HTTP call to your check, but does not wait for an answer. Once your check reaches a decision, it fires a callback to communicate it. Azure DevOps resumes your pipeline’s execution.

Two settings relevant for checks’ scalability are Time between evaluations and Timeout, both under Control options. Time between evaluations defines how long a check’s decision is valid. A value of 0 means the decision is final. A non-zero value means the system retries the check after the configured interval, when its decision is negative or there are other approvals and checks. Timeout defines how long Azure DevOps waits for a positive decision. Together, time between evaluations and timeout define how many times the system can try a check.

Your checks are considered compliant when they follow these rules depending on the mode you use and the number of retries:

  • ApiResponse mode (or Synchronous mode): You limit the number of retries for your check to 10. You can do this in a number of ways. For example, you can configure timeout to 20 and time interval between evaluations to 2. Or, you can configure timeout to 100 and time interval between evaluations to 10. But, if you configure timeout to 100 and set the time interval between evaluations to 2, you are essentially asking Azure Pipelines to retry your check for a maximum of 50 times, and it is therefore not compliant. To summarize, the ratio of timeout to time interval between evaluations should be less than or equal to 10.
  • Callback mode (or Asynchronous mode): In this case, Azure Pipelines does not retry your checks, and you are expected to provide a result asynchronously whenever you consider the result of your evaluation to be final. For this to be considered compliant, you must configure time interval between evaluations to 0.

There is no need to retry a Callback check. You fully control your check’s logic, so you know best if and when a pipeline run has met the criteria to proceed with its execution. There is no need for Azure DevOps to probe your Azure Function check every 15 minutes. All Azure DevOps needs is a decision within the configured Timeout. The decision must be final, be it fail or pass.

If your checks follow our recommended usage, they are compliant.

When Azure DevOps evaluates an Invoke Azure Function or REST API check, it makes an HTTP call. The check fails if the call does not complete in 3 seconds.

Rollout

Our goal is that by the end of the year, all your Invoke Azure Function & REST API check are compliant. You may need to take action to ensure this.

We realize this change is disruptive, so we have a three-phase rollout plan:

Phase 1

Starting June:

  • When you create a check, its settings are compliant, but you can still edit them to make them non-compliant
  • Azure DevOps shows warnings for non-compliant checks encouraging you to update their settings

We show warnings at resource level, for example, this is how it looks like for the FabrikamFiber Feed WUS1 environment, which has non-compliant checks configured. Warnings on resources with non-compliant checks

When you open a resource’s details page, we show warnings on the Approvals and checks tab for non-compliant checks. checks phase 2 environment warnings

Finally, when you click on a check, we focus the input on the setting you need to update. Screenshot showing warnings in the configuration side panel for an Invoke Azure Function

Phase 2

Starting July 24th:

  • You can only create compliant checks
  • When you edit a check, you can save changes only if its settings are compliant
  • Azure DevOps shows more prominent warnings for non-compliant checks encouraging you to update their settings

In addition to the warnings in Phase 1, we show warnings in a pipeline run’s detail page.

We show warnings in the Stages card. For example, this is how it looks for a stage named Deployment FabrikamFiber Feed WUS1 that accessed resources protected by non-compliant checks. Screenshot showing warnings in a pipeline run's details page.

If you open the Approvals and checks side panel, we show warnings next to the non-compliant check. Warnings phase 2 checks panel

Phase 3

Starting Autumn:

  • Weekly, one-day long brownouts, where all pipeline runs that use non-compliant checks fail
  • Starting Winter, all pipeline runs that evaluate non-compliant checks fail

A pipeline run that failed due to non-compliant checks shows an error in the Errors card guiding you to this blogpost. Screenshot of a failed pipeline run due to non-compliant checks

Experience improvements

We are making reading check logs easier. Checks are critical sources of information for your deployment’s success. They can tell you if you forgot to close a work item ticket, for example. Alas, knowing that a check provided such critical information is hard today. The pipeline run details page shows the latest check log only for checks that follow our recommended usage.

The pipeline run details page showing check logs for compliant checks

To prevent mistakenly approved Approvals, Azure DevOps shows the Instructions to approvers in the Approvals and checks side panel in a pipeline run’s detail page.

approval instructions in checks panel

FAQ

How do I make sure my checks continue to work?

If you followed our guideline on recommended usage, your checks need no further changes.

Can I still use synchronous Invoke Azure Function or REST API checks?

Yes, but there is a 10-try limit.

My Azure Function takes more than 3 seconds to start up and process the request. What can I do?

If the HTTP call Azure DevOps makes to your check returns an HTTP status code outside of the 2xx and 3xx range, it retries for a maximum of 10 times. Then, the check fails. This is a built-in retry by the system that you do not have to configure anywhere.

I don’t see the check logs in my pipeline run details page. Why?

The pipeline run details page shows the latest check log only for checks that follow our recommended usage.

Silviu Senior Product Manager, Azure DevOps

Read next

11 comments

Discussion is closed. Login to edit/delete existing comments.

  • Henryk Scheibner 1

    So this means we can no longer emulate the “defer deploy” feature classic pipelines had unless we run a service that can actively call back azure devops later, or are there plans to implement a similar feature for yaml pipelines?
    (We can not use the business hours restriction on an environment since we may need to overwrite that for urgent fixes/new application deploys and it can’t be removed without re-running the entire multi stage pipeline).

    • SilviuMicrosoft employee 1

      Hi Henryk,

      You are correct, you will have to run a service that calls back Azure DevOps.

      The classic release pipelines feature you’re referring to is the “Approval with Defer deployment for later” one, right? We are working toward reducing the feature gap between classic release pipelines and YAML pipelines, and this feature is on our radar.

      Best regards,
      Silviu

      • Henryk Scheibner 0

        Hi Silviu,

        Yes, I mean the manual user approval with the checkbox “Defer deployment for later” the classic pipelines have.
        We don’t need much complex logic just manual (granular) control over the timing right now.
        I’m glad to hear the feature is on your radar. Do you know if it’ll be available before this changes takes effect?
        Unless we have to I’d rather not convert our pipelines back to classic pipelines or spin up a web service with a reliable timer behind it.

        Best,
        Henryk

        • SilviuMicrosoft employee 0

          Hi Henryk,

          To minimise risks to your pipelines, I would advise you to go ahead and spin up a web service with a reliable timer behind it.

          Best regards,
          Silviu

      • John Townsend 0

        Hi Silviu,

        I know I’m a bit late to this conversation, but this is something that my team and I would greatly benefit from. Of the initiatives on the roadmap, I’m not seeing this one listed.

        https://learn.microsoft.com/en-us/azure/devops/release-notes/features-timeline

        What’s the best way I can advocate for parity of this feature to be expedited?

        Thanks,
        John

  • Flemming Seerup 0

    Hi – is it possible to have Environment setup with Approvals for a group/user, but have some pipelines which can run without manual approval ?
    I am trying to run a scheduled pipeline daily doing some sanity check, but some of our environments require approvals, and I can’t seem to find a workaround for this.

    • SilviuMicrosoft employee 0

      Hi Flemming,

      No, it’s not possible. Approvals are a security mechanism, so they are enforced for all pipelines.

      Best regards,
      Silviu

  • Andrew Barger 0

    There is a feature gap that makes this change very disruptive. We advised our developers to make changes to support the new restrictions with the expectation that timed-out stages could be retried. But we find that once these stages are skipped, they cannot be retried. See https://developercommunity.visualstudio.com/t/re-run-skipped-stages-1/1076765

    We have contacted our Microsoft representative about this, but I also wanted to provide some direct feedback because this is going to hit us really hard. We have thousands of pipelines impacted by this change.

  • Jonas Samuelsson 0

    What scopes are required for accessing the callback endpoint using my own PAT?
    I haven’t been able to find that in any documentation.

    • SilviuMicrosoft employee 0

      Hi Jonas,

      To call into Azure DevOps, we recommend you use the job access token issued for the check execution, instead of a personal access token (PAT). The token is already provided to your checks by default, in the AuthToken header. You can read about its benefits in our Invoke Azure Function / REST API checks doc.

      If you must use a PAT, then, for the time being, it must be a full-access one.

      Best regards,
      Silviu

Feedback usabilla icon