Replacing “View YAML”

Matt

This sprint, we’re replacing the “View YAML” experience. This is the feature which helps you migrate designer pipelines to YAML. The new version is more correct and covers more Classic Build features, which I’ll cover in this post. It removes one useful quirk of the old implementation, so I’ll share tips for anyone who depended on that quirk.

The old experience

The previous version of this feature worked on a job or step at a time. If you’ve never used it before, you can see how the old experience worked below.

Once the new experience ships, if you click that button, you’ll be greeted with a message telling you that the feature has moved.

View yaml has moved

The new experience

Following the directions in the dialog box, let’s see how the new experience works. First, we’ll export our pipeline as YAML.

The Classic pipeline is left intact and nothing changes. We get a copy of the pipeline, expressed in YAML.

The exported YAML

More correct

The new implementation is more correct. Previously, the feature worked entirely at the web app layer. Using a combination of editor features and translations of platform code, it built a YAML representation of whatever was currently loaded in the editor.

As many users can attest, the “View YAML” experience was sometimes wrong. The differences were typically subtle, small, and hard to spot. Although we’ve fixed many of them over the years, we never gained confidence that we had correctly handled every edge case. We’ve even been known to (accidentally) break what had previously been working, such as when we added new YAML keywords.

The new experience takes a different tactic. It’s implemented in the platform, reusing existing Classic and YAML pipeline infrastructure. This lets us reuse instead of repeating code, for instance our YAML parsing library.

Using a clever testing strategy, we can be much more confident that the implementation is correct today and remains correct in the future. The tests consist of actual Classic pipelines which we run through the designer to YAML conversion process. Then, both the original Classic and resulting YAML pipelines are compiled into their runtime format, as if we were going to schedule them. Finally, we compare the plans – if they’re identical, then we know the conversion was completely lossless.

Covers more features

In addition to being more correct, the new version covers more Classic pipeline features. The old experience couldn’t even export more than one job at a time. And while we could have imagined beefing up the old experience, it would have been subject to the same correctness problems described above.

Building the new version, we stepped back and thought about the problem more systematically. We used the UI and the existing YAML parser as our starting points, working to cover each area. The new system knows how to handle every feature listed here:

  • Single and multiple jobs
  • Checkout options
  • Execution plan parallelism
  • Timeout and name metadata
  • Demands
  • Schedules and other triggers
  • Pool selection, including jobs which differ from the default
  • All tasks and all inputs, including optimizing for default inputs
  • Job and step conditions
  • Task group unrolling

In fact, there are only two areas which we know aren’t covered.

Variables

Variables defined in YAML “shadow” (hide) variables set in the UI. Therefore, we didn’t want to export them into the YAML file in case you need an ability to alter them at runtime. If you have UI variables in your Classic pipeline, we mention them by name in the comments to remind you that you need to configure them in your new YAML pipeline definition.

Timezone translation

cron schedules in YAML are in UTC, while Classic schedules are in the organization’s timezone. Timezone handling has a lot of sharp edges, so we opted not to try to be clever. We export the schedule without doing any translation, so your scheduled builds might be off by a certain number of hours unless you manually modify them. Here again, we make a note in the comments to remind you.

Easily editing YAML

As great as the new design is, it takes away one use case. Fortunately, we offer a solid replacement.

If you don’t typically edit YAML pipelines, it can be challenging to get started. We know many of you would create a new Classic pipeline, tweak the settings for a task or two, and then use “View YAML” as a sort of visual editor for YAML pipelines.

For this use, we encourage you to check out the YAML pipeline editor and its built-in Task Assistant. The Task Assistant is a pane on the right side of the screen which helps you correctly create and modify YAML steps. Unlike the “View YAML” button, it works directly in the real YAML file. Go to a YAML pipeline definition and choose “Edit”; you’ll be dropped into a web-based editing experience. From there, open up the Task Assistant and start editing away.

In closing

We hope you enjoy the new, more correct and complete replacement for “View YAML”.

31 comments

Comments are closed. Login to edit/delete your existing comments

  • Krzysztof Madej

    Cool thanks, that’s great and warmly expected by community.!Are you going to extend that also for Release pipelines?

    • Matt CooperMicrosoft employee

      No plans to do so. Classic RM pipelines are different enough in their execution that I can’t make the same strong guarantees about correctness as I can with classic Build. Also, a number of concepts were re-thought between RM and unified YAML pipelines. In some cases, there isn’t a direct translation for an RM feature. A human is required to think about what the pipeline is designed to accomplish and re-implement it using new constructs.

      • Krzysztof Madej

        Thanks for replying. And since it is not planned for releases I reported a bug becuase the same message appears on Releases when you click View YAML.

          • Jeffrey Patton

            Will this bug be addressed and corrected? or dropped and ignored? we use the view yaml feature in releases now and the lack of it is frustrating to say the least.

          • Matt CooperMicrosoft employee

            @Jeffrey Patton – it will be fixed. (Sorry, I can’t reply to your comment because apparently we’re nested too deeply!)

          • Tolga Kavukcu

            I would like to give feedback on that, this is blocking our work as we need to move some release pipelines to YAML. When should we expect that to be fixed? It’s frustrating for us too

          • Tim Katje

            So for clarity’s sake, since I cannot tell from this comment chain or the referenced bug, does Microsoft consider the fact that the button is now broken in Classic Releases the bug, and we will be getting that functionality back, or will the button simply be removed?

            It’s always funny/ironic when the one-time I need a simple feature like this, it’s right after a change unintentionally kills it. Or at least, I hope it’s unintentional and that it’ll be back. I’ve got quite a few conversions planned going into next year…

          • Hans Goossen

            I’m as well blocked by this issue.
            It’s a real shame that you are leaving us hanging for 2 weeks already. Also your QA process leaves a great deal to be desired.

      • Heigo Niilop

        What about Task Group? There is also “View YAML” and now it is not possible to see task group tasks yaml.

        • Ben Sgroi

          We’re having the same issue. “Export to YAML” does not appear to be unrolling task groups, and now there’s no way to export all of those task groups to YAML, short of manually attempting to recreate them in YAML. That can be a lot of extra effort if you have a lot of task groups :-/

          • Bratton, Robert

            A generic JSON to YAML converter won’t work because there are Azure DevOps specific items in the YAML pipelines that they won’t understand.

  • Davy King

    Is this going to ship in any updates in Azure DevOps Server 2020?

  • Dhawal ParkarMicrosoft employee

    This is great ! Is it possibly to export to yaml using an API ? We have a lot of pipelines !

    • Matt CooperMicrosoft employee

      Yes, there’s a REST API – though in trying to get you the exact route, I see we have a bug. Right now it seems you have to append “?dummyValue=” to the definition API to get back YAML, but that’s definitely not what we intended to do.

      https://dev.azure.com/{ORG}/{PROJECT}/_apis/build/definitions/{BUILD-ID}?dummyValue=
      • Seifeddine Dridi

        The YAML preview feature doesn’t actually work when there are resources in the pipeline. Is this a known issue or should I submit a bug?

  • Owen Davies

    Nice,

    Is there an upcoming feature to allow the creation of pipelines by simply creating the yaml in the repo and committing? ala github actions?

    • Matt CooperMicrosoft employee

      I believe if you create a file “azure-pipelines.yml” or “.azure-pipelines.yml” in an Azure Repos Git repository, we automatically create a pipeline from that.

  • Matt CooperMicrosoft employee

    Someone left us feedback through an anonymous system where I don’t have the ability to reply. Hoping they’ll see my answer here!

    My company and other companies need a LTS statement on how long YAML will be supported for build pipelines.

    YAML is where all of our investment in Azure Pipelines has gone for several years now. We’ve heard nervousness from customers about whether classic Pipelines would ever go away (spoiler alert: nope) but this is the first time I’ve heard the same about YAML. While I can’t predict the distant future, I can tell you that we have no plans to deprecate YAML pipelines and they’re considered state of the art for Azure DevOps.

  • Dan McHugh

    This is awesome. We have a bunch of pipelines that use custom tasks. For these pipelines when we export the YAML the file is truncated (ends with …) and doesn’t contain the YAML for the underlying tasks. The JSON that is export is complete.

    Will custom tasks be supported for the YAML export (since it appears fine in the JSON export)?

    In the meantime, is there a suggested workaround to convert the JSON into a YAML pipeline?

    • Matt CooperMicrosoft employee

      Hi Dan, it should work with custom tasks just the same as it works with in-box tasks. If you want to send me the YAML and JSON exports of your pipeline, I can take a look. I’m mattc@xbox.com.

      The ... thing is silly but as it turns out, that’s a valid way to end a YAML file. Our YAML library did it by default so we left it in, but I see how it might look like the file is truncated. We’ll look at suppressing that to reduce confusion.

  • James O'Neill

    Sorry to say it but the way this has been done is dreadful

    1. There is still a link there but it does not do the job. It should have had period of showing the yaml and with a message saying “This is moving – you can find at its new home …. ” for a while . There was no need to ever stop the old way from working, you could have just added the new one.
    2. “The Runs page” Nothing I’ve ever seen calls it the runs page. I only got to this blog because I’d wasted 10 minutes trying to figure out which page was meant, I don’t think I’ve ever seen the main page for the pipeline called the “runs page” because it has runs / branches / analytics on it.
    3. Where before you could get the settings for a single task (I wanted to get the parameters for one task as a text block not a bitmap screen shot) Now you can only download the yaml for the whole pipeline (20+ tasks in my case)

    There must have been 100+ useful things that you could have spent time, but you put effort into this which takes away something which worked . How to lose friends and alienate people.

    • James O'Neill

      Supplemenatary – the old method still works for a release pipeline, so there are two methods for obtaining the Yaml. If looking at a release one uses the old method but if looking at a build one has to use the new one.

    • David V. Corbin

      100% agreement (see my other comment for additional reasons this was a poor implementation change.)

  • David V. Corbin

    That will be helpful to many…. but also seems like a breaking change (there is tooling used in real client environments that counts on the existing behavior). To the best of my knowledge there is no way to “pre-detect” which experience is going to occur, making it difficult (at best) to update these types of tools…… Was there NO feasible way to do this such that it did not break existing experience???

  • Michael Cronqvist

    When using the classic release pipeline you can setup “Report deployment status to Boards”. Are there any plans to support this work item deployment tracking feature when using YAML pipelines?