You can’t have security for DevOps until you have DevOps for security
Of all the tenets of DevOps, the one that really stands out for me as a game-changer is its focus on lean product development. Lean product development tells us that the most successful organizations are the ones who can most quickly iterate on the problem hypothesis -> minimum viable product (MVP) solution -> feedback loop. Let’s say you get an insight into a customer pain point while checking your mail over your morning coffee. If you can execute quickly enough to deploy a quick-and-dirty experimental solution to a test ring of customers by lunchtime and start analyzing solution telemetry by the time you’re closing up for the day, you’re going to win. You might not get the solution right the first time or even the third time, but if your iteration loop is only 24 hours long, you’re going to learn quickly and get the solution right much faster than if you use a heavyweight six-month waterfall approach.
We need to apply the same principle to secure development – that is, rapidly iterating on the actual set of activities that developers complete to ensure the code they’re building is secure. The faster we can iterate on refining secure development practices, the faster our developers will be able to address security pain points, and the better we’ll protect our customers. Many people have asked the question, “How can we bring security practices into DevOps?” but only a few have asked “How can we bring DevOps practices to secure development?” This second question is key. You can’t have security for DevOps until you have DevOps for security. Here’s a straightforward framework that we’ve used on the One Engineering System (1ES) Security team with excellent results:
- First, determine a SMART, business-goal-oriented OKR to aim for
- Next, develop and validate problem hypotheses. In other words, find out why you’re not already achieving the objective that you want
- Finally, develop and validate solution hypotheses (aka MVP features) to solve the specific problem you validated
DevOps for security in practice
Laura is responsible for ensuring that no products in her organization use open source components with known common vulnerabilities and exposures (CVEs). She sets an OKR for her organization to improve their fix times for identifying and upgrading references to vulnerable packages by 20% this quarter. Based on a recent experience with a product team, she thinks that a top root cause of slow fix times is that upgrading packages causes compatibility problems that need to be regressed. But when she talks with more teams and analyzes some telemetry data, she finds that compatibility regressions are rare. She then hypothesizes that slow fix times happen because developers only get alerted to vulnerability problems late in the DevOps lifecycle, after code has already shipped. This time, her customer development team interviews and data analysis confirm her hypothesis.
Laura and her security engineering team quickly design a rough IDE plug-in that alerts developers when they first include a reference to a new vulnerable package. They deploy the plug-in to some pilot teams and keep a close eye on these teams’ fix rate telemetry over the next few days. The telemetry shows a 30% reduction in fix times – even better than they aspired to – so they deploy the plug-in across their entire org and celebrate their success.
This kind of targeted, rapid, lean approach to security has worked wonders in the 1ES team at Microsoft. For just one recent success story, last year a security researcher privately disclosed to us a new critical supply-chain vulnerability. After some quick analysis, we discovered that thousands of Microsoft teams were vulnerable to this newly discovered problem, and we had to ensure every one of them was fixed before the finder started the public disclosure phase. Within days we had validated a problem hypothesis and deployed an MVP solution that detected vulnerable builds and alerted the affected teams. However, it became clear from our telemetry that this MVP wasn’t moving the needle quickly enough. We pivoted to a stronger solution that broke affected builds rather than just alerting, and soon everyone had fixed their vulnerabilities.
Learn more
If you’d like to learn more about the principles of lean customer development and how you can put them into practice in your own organization, I recommend Lean Customer Development: Building Products Your Customers Will Buy by Cindy Alvarez. While it’s not specifically focused on security, the approach that Cindy presents in her book is widely applicable across disciplines and has worked well for us in 1ES Security. I encourage you to try it for yourself.
Want to come work with us? Check out https://aka.ms/1esjobs for open positions.
Light
Dark
0 comments