{"id":163,"date":"2021-09-16T10:04:44","date_gmt":"2021-09-16T17:04:44","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/engineering-at-microsoft\/?p=163"},"modified":"2021-09-16T10:04:44","modified_gmt":"2021-09-16T17:04:44","slug":"you-cant-have-security-for-devops-until-you-have-devops-for-security","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/engineering-at-microsoft\/you-cant-have-security-for-devops-until-you-have-devops-for-security\/","title":{"rendered":"You can\u2019t have security for DevOps until you have DevOps for security"},"content":{"rendered":"

Of all the tenets of DevOps<\/a>, the one that really stands out for me as a game-changer is its focus on lean product development<\/a>. Lean product development tells us that the most successful organizations are the ones who can most quickly iterate on the problem hypothesis -> minimum viable product (MVP) solution -> feedback loop. Let\u2019s say you get an insight into a customer pain point while checking your mail over your morning coffee. If you can execute quickly enough to deploy a quick-and-dirty experimental solution to a test ring of customers by lunchtime and start analyzing solution telemetry by the time you\u2019re closing up for the day, you\u2019re going to win. You might not get the solution right the first time or even the third time, but if your iteration loop is only 24 hours long, you\u2019re going to learn quickly and get the solution right much faster than if you use a heavyweight six-month waterfall approach<\/a>.<\/p>\n

We need to apply the same principle to secure development \u2013 that is, rapidly iterating on the actual set of activities that developers complete to ensure the code they\u2019re building is secure. The faster we can iterate on refining secure development practices, the faster our developers will be able to address security pain points, and the better we\u2019ll protect our customers. Many people have asked the question, \u201cHow can we bring security practices into DevOps?\u201d but only a few have asked \u201cHow can we bring DevOps practices to secure development?\u201d<\/em> This second question is key. You can\u2019t have security for DevOps until you have DevOps for security. <\/strong>Here\u2019s a straightforward framework that we\u2019ve used on the One Engineering System (1ES) Security team with excellent results:<\/p>\n

    \n
  1. First, determine a SMART<\/a>, business-goal-oriented OKR<\/a> to aim for<\/li>\n
  2. Next, develop and validate problem hypotheses. In other words, find out why you\u2019re not already achieving the objective that you want<\/li>\n
  3. Finally, develop and validate solution hypotheses (aka MVP features<\/a>) to solve the specific problem you validated<\/li>\n<\/ol>\n

    DevOps for security in practice<\/h3>\n

    Laura is responsible for ensuring that no products in her organization use open source components with known common vulnerabilities and exposures (CVEs<\/a>). She sets an OKR for her organization to improve their fix times for identifying and upgrading references to vulnerable packages by 20% this quarter. Based on a recent experience with a product team, she thinks that a top root cause of slow fix times is that upgrading packages causes compatibility problems that need to be regressed. But when she talks with more teams and analyzes some telemetry data, she finds that compatibility regressions are rare. She then hypothesizes that slow fix times happen because developers only get alerted to vulnerability problems late in the DevOps lifecycle, after code has already shipped. This time, her customer development team interviews and data analysis confirm her hypothesis.<\/em><\/p>\n

    Laura and her security engineering team quickly design a rough IDE plug-in that alerts developers when they first include a reference to a new vulnerable package. They deploy the plug-in to some pilot teams and keep a close eye on these teams\u2019 fix rate telemetry over the next few days. The telemetry shows a 30% reduction in fix times \u2013 even better than they aspired to \u2013 so they deploy the plug-in across their entire org and celebrate their success. <\/em><\/p>\n

    This kind of targeted, rapid, lean<\/em> approach to security has worked wonders in the 1ES team at Microsoft. For just one recent success story, last year a security researcher privately disclosed to us a new critical supply-chain vulnerability. After some quick analysis, we discovered that thousands of Microsoft teams were vulnerable to this newly discovered problem, and we had to ensure every one of them was fixed before the finder started the public disclosure phase. Within days we had validated a problem hypothesis and deployed an MVP solution that detected vulnerable builds and alerted the affected teams. However, it became clear from our telemetry that this MVP wasn\u2019t moving the needle quickly enough. We pivoted to a stronger solution that broke affected builds rather than just alerting, and soon everyone had fixed their vulnerabilities.<\/p>\n

    Learn more<\/h3>\n

    If you\u2019d like to learn more about the principles of lean customer development and how you can put them into practice in your own organization, I recommend Lean Customer Development: Building Products Your Customers Will Buy<\/a><\/em> by Cindy Alvarez. While it\u2019s not specifically focused on security, the approach that Cindy presents in her book is widely applicable across disciplines and has worked well for us in 1ES Security. I encourage you to try it for yourself.<\/p>\n

    Want to come work with us?<\/strong> Check out https:\/\/aka.ms\/1esjobs<\/a> for open positions.<\/p>\n

     <\/p>\n

     <\/p>\n

     <\/p>\n","protected":false},"excerpt":{"rendered":"

    The faster we iterate on refining secure development practices, the faster our developers can address security pain points, and the better we protect our customers. In this post, Bryan Sullivan walks through key learnings from the 1ES Security team.<\/p>\n","protected":false},"author":64455,"featured_media":170,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[4,7,8,2],"class_list":["post-163","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engineering-at-microsoft","tag-1es","tag-devops","tag-lean-product-development","tag-security"],"acf":[],"blog_post_summary":"

    The faster we iterate on refining secure development practices, the faster our developers can address security pain points, and the better we protect our customers. In this post, Bryan Sullivan walks through key learnings from the 1ES Security team.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/engineering-at-microsoft\/wp-json\/wp\/v2\/posts\/163","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/engineering-at-microsoft\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/engineering-at-microsoft\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/engineering-at-microsoft\/wp-json\/wp\/v2\/users\/64455"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/engineering-at-microsoft\/wp-json\/wp\/v2\/comments?post=163"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/engineering-at-microsoft\/wp-json\/wp\/v2\/posts\/163\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/engineering-at-microsoft\/wp-json\/wp\/v2\/media\/170"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/engineering-at-microsoft\/wp-json\/wp\/v2\/media?parent=163"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/engineering-at-microsoft\/wp-json\/wp\/v2\/categories?post=163"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/engineering-at-microsoft\/wp-json\/wp\/v2\/tags?post=163"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}