Let’s Hack a Pipeline: Argument Injection
Welcome to Let’s Hack a Pipeline! In this series of posts, we’ll walk through some common security pitfalls when setting up Azure Pipelines. We don’t really want to get hacked, so we’ll also show off the mitigation.
Episode I is titled Argument Injection. Episode II and Episode III are now also available.
Preface on security
A quick note before we begin: security is a shared responsibility. Microsoft tries very hard to set safe, sensible defaults for features we deliver. Sometimes we make mistakes, and sometimes threats evolve over time. We have to balance security with “not breaking people’s things”, especially for developer tools. Plus, we can’t control how customers use the features we build. The purpose of this series is to teach the problems, which we hope helps you avoid them in practice.
The setup
Here’s a pretty standard pipeline for running a build tool like MSBuild. We’ve left out common steps like “restore NuGet packages” and “run tests” to focus on the vulnerable step.
pool: { vmImage: 'windows-latest' }
steps:
- script: >
msbuild App1.sln
/p:platform="$(platform)"
/p:configuration="$(configuration)"
Now, there’s nothing really MSBuild-specific about this attack. It’s a convenient target since it accepts so many parameters and inputs, but I could do this with basically any tool. Two of those inputs are variables stored with the pipeline: platform and configuration.
Normally these would be values like x64 and Release. They’re editable at queue time in case someone wants to run a different flavor, like x86 and Debug. At runtime, Azure Pipelines interpolates the script step with the values of variables surrounded by $( ). The MSBuild command becomes: msbuild App1.sln /p:platform="x86" /p:configuration="Release".
The attack
I’m not limited to Release and Debug as options. When I queue the pipeline, I can put pretty much whatever I want into those variables. Like:
By setting configuration to Debug" & echo Villain was here > file.txt &::, I make the MSBuild command read:
msbuild App1.sln /p:platform="x86" /p:configuration="Debug" & echo Villain was here > file.txt &::"
^ <--- contents of the variable --> ^
That line says “run MSBuild, then write some stuff to a file, then skip the rest of the line”. (The double colon is used to comment out the trailing double-quote the base script contains.)
OK, this payload is not very sophisticated. A lot of them initially aren’t. But I bet you can dream up something more nefarious to do than leaving a calling card.
Why this works
Much like SQL injection, the attacker abused what should have been an argument to the desired function. The attacker completed the argument prematurely (the Debug" part – note the ") and added an additional command. Once Azure Pipelines interpolated the variables, the script was perfectly legal but malicious.
Azure Pipelines variables are strings. Since they can be used for a variety of purposes, the system can’t quote, escape, or otherwise mangle them. That must be arranged by the code accepting the input.
Mitigating argument injection
There are two ways to mitigate this problem:
- runtime parameters
- tool-specific tasks
Runtime parameters
The general and preferred option is to use runtime parameters. You can limit runtime parameters to certain, expected values. When someone queues the build, their options are constrained and they can’t write an argument injection attack.
parameters:
- name: platform
type: string
values:
- x86
- x64
- name: configuration
type: string
values:
- Debug
- Release
pool: { vmImage: 'windows-latest' }
steps:
- script: >
msbuild App1.sln
/p:platform="${{ parameters.platform }}"
/p:configuration="${{ parameters.configuration }}"
In other scenarios, runtime parameters offer even more help. They have data types like number and boolean. This is helpful when an argument expects only integers, for example.
Tool-specific tasks
Another mitigation is specific to the tool you’re using. Instead of a script, if this pipeline had used the MSBuild or VSBuild tasks, inputs are escaped properly and rejected if they contain characters like ". Escaping and illegal characters vary by tool, which is why the task must be specific to the tool.
pool: { vmImage: 'windows-latest' }
steps:
- task: MSBuild@1
inputs:
solution: App1.sln
platform: $(platform)
configuration: $(configuration)
If you’re a task author and want to see how we do this in the built-in tasks, take a look at the MSBuild task and its helper code.
Review
Queue-time variables offer no quoting, escaping, or guarantees about their contents. A malicious person with the right to queue builds can inject their own commands into script arguments or vulnerable tasks.
Use runtime parameters to limit what inputs will be accepted. For additional protection, use purpose-built tasks which properly escape (or reject) malicious inputs.
Light
Dark
7 comments
This misses the point… You’re building my code! There are so many, many ways I can “hack” you because I will always find a way to run custom code. Say what if I add an MS build step in my project with my custom tool?
To me, it’s rather all about access management and secret rotation. Unfortunately DevOps kind of sucks at this. Permissions are a mess. For example you can’t allow someone to queue builds without being able to see pipeline yaml. You also can’t let someone see code-based wiki without granting read access to the enire repo.
I tend to agree with Jonathan here. If you you’re concerned with compiling my source code, just wait till you see what the compiled code does…
A better approach to securing a pipeline would be focusing on what resources the pipeline can access, who can run the pipeline and in what queue/pool. We can set access contorol on all sorts of things in the devops space including pools and secrets.
Did the pipeline run in a container/temp vm? That would be a good step. If you’re not, you really need to focus even more on what the agent has access to on the box.
To a person used to Bash scripts, this looks weird! Anything inside double quotes should be escaped to make sure it ends up as a double quoted string. Why would one use quotes around an interpolated string if it may not end up as a single string?
I don’t follow. The string gets interpolated before Bash ever gets a chance to run. All Bash sees is the final result, which is why this issue exists. The YAML parser doesn’t know anything about Bash, PowerShell, or any other tools. So it should definitely not be in the business of escaping/quoting strings.
I have reported similar behaviour to MS security team around 6 months ago. Their reply was that this is not a security related issue, but this was “just a bug”.
This post describes one way that a (non-buggy) feature can be used insecurely. I don’t know the specifics of your report so I don’t know whether MS security made a mistake. If you’d like, you can ask them to escalate it to me.
Getting 404s on the two links in Tool-specific tasks: “MSBuild task” and “helper code”