{"id":59645,"date":"2020-08-27T05:41:04","date_gmt":"2020-08-27T13:41:04","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/devops\/?p=59645"},"modified":"2020-08-27T05:41:04","modified_gmt":"2020-08-27T13:41:04","slug":"pipeline-shared-infrastructure","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/devops\/pipeline-shared-infrastructure\/","title":{"rendered":"Let’s Hack a Pipeline: Shared Infrastructure"},"content":{"rendered":"

Welcome back to Let’s Hack a Pipeline<\/em>.\nWe’ve seen argument injection<\/a> and source code stealing<\/a>.\nThis week, we’ll wrap up the miniseries with Episode III: a Shared Infrastructure<\/strong> attack.<\/p>\n

One more time: security is a shared responsibility.\nThe purpose of this series is to showcase some potential pitfalls to help you avoid them.<\/p>\n

The setup<\/h2>\n

Let’s say I’m part of a large company called Fabrikam.\nFabrikam’s Azure DevOps organization is divided into lots of separate projects.\nWe have a centralized team responsible for setting up pipelines infrastructure.\nThe central team has created an agent pool full of powerful build machines called FabrikamPool<\/code>.\nFabrikamPool<\/code> is shared with several projects.\nThis way, every team has access to these beefy machines.<\/p>\n

The attack<\/h2>\n

This isn’t so much “an” attack as a class of possible attacks.\nAll I have to do is compromise one pipeline in any project.\nI could be an outside attacker or even a malicious insider.\nMaybe I use one of the attacks mentioned in previous Let’s Hack a Pipeline posts.<\/p>\n

From there, I can do anything to the agent that its credentials allow.\nI can also change anything on the host machine that the agent has access to.\nMaybe I install a persistent backdoor which lets me remote into the machine.\nOr, maybe I install a filesystem watcher that can automatically steal code the next time a pipeline runs from another project.\nWhat about a hacked compiler that adds malicious code to everything it compiles?\nOnce I’ve poisoned the agent, my attack possibilities are basically endless.<\/p>\n

Why this works<\/h2>\n

When you run a pipeline job, you’re extending the trust boundary out to a machine that’s beyond Azure DevOps’s direct control.\nIf multiple projects are each targeting the same agent instance, then everyone is at the mercy of the least-defended pipeline.\nAnd most self-hosted agents are persistent, meaning that the environment they run on lasts beyond the scope of a single job.<\/p>\n

The agent segregates runs from different pipelines into their own folders, but that’s for convenience, not secure isolation.\nIn fact, there’s a whole class of non-malicious, but still painful, “poisoning” that can take place.\nImagine a pipeline which needed to test changing some operating system-level feature (TLS 1.2, localization) or installing a global package (new version of Python).\nAll the other pipeline jobs which run on that agent would see the changed environment, whether or not they expected it.<\/p>\n

Mitigating attacks on infrastructure<\/h2>\n

The best way to mitigate this attack is to not share infrastructure.\nProjects are a pretty firm security boundary, and shared pools violate that boundary.<\/p>\n

One-time use agents<\/h3>\n

For any pipelines where you can, you should prefer using Microsoft-hosted or scale set agents with one-time use agents.\nBecause the agent is reimaged after each use, there’s no standing or persistent infrastructure to attack.<\/p>\n

Minimal permissions<\/h3>\n

If you can’t use Microsoft-hosted or scale set agents, run your self-hosted agent software with the minimal privileges needed to run your pipelines.\nRemember, the agent’s job is to run arbitrary, untrusted code (that’s what CI does<\/em>, after all).\nYour host machine should treat the agent as potentially malicious.<\/p>\n

Malicious pipelines in the same project<\/h3>\n

Security exists in balance with other attributes like usability and maintainability.\nWhile it would increase security to maintain a dedicated pool of agents per-pipeline, this would likely be expensive, hard to use, and hard to manage.\nWe think that the project level is an appropriate middle ground: you get a fair amount of isolation while still keeping most of the benefits of shared infrastructure.\nYour security team may feel differently, and you should absolutely consider other spots along this spectrum.<\/p>\n

Review<\/h2>\n

Sharing pools across projects is tempting since you can potentially save money and complexity by setting up fewer agents.\nHowever, that configuration puts all pipelines in all projects at risk.\nThe most insecure pipeline in any project becomes your best-case scenario.<\/p>\n","protected":false},"excerpt":{"rendered":"

Welcome back to Let’s Hack a Pipeline. We’ve seen argument injection and source code stealing. This week, we’ll wrap up the miniseries with Episode III: a Shared Infrastructure attack. One more time: security is a shared responsibility. The purpose of this series is to showcase some potential pitfalls to help you avoid them. The setup […]<\/p>\n","protected":false},"author":719,"featured_media":45953,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[224],"tags":[],"class_list":["post-59645","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure"],"acf":[],"blog_post_summary":"

Welcome back to Let’s Hack a Pipeline. We’ve seen argument injection and source code stealing. This week, we’ll wrap up the miniseries with Episode III: a Shared Infrastructure attack. One more time: security is a shared responsibility. The purpose of this series is to showcase some potential pitfalls to help you avoid them. The setup […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/59645","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/users\/719"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/comments?post=59645"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/posts\/59645\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media\/45953"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/media?parent=59645"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/categories?post=59645"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/devops\/wp-json\/wp\/v2\/tags?post=59645"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}