DevOps Stories – An Interview with Brian Blackman of Microsoft Premier
App Dev Manager Dave Harrison talks with Brian Blackman, a Senior Consultant for Microsoft Premier. As an Azure development and DevOps consultant, he works with Independent Software Vendors (ISV), Enterprises, and Partners where his specialty is DevOps. Brian’s focus is assisting customers in establishing a vision for DevOps, assessing where they currently stand, and then working with them to develop a roadmap to attain their vision. MCTP, MCSD-C++, CSM, CSP, SAFe Agilist.
Note – these and other interviews and case studies will form the backbone of our upcoming book “Achieving DevOps” from Apress, due out in late 2018. Please contact me if you’d like an advance copy!
My story – well, I came from the OEM division, that’s really hands on – including working with Windows Mobile, Windows for Automotive as a program manager. It was great stuff; very close to the metal, working right with automotive systems from a hardware / software interface point of view. At Microsoft Consulting Services (MCS) I got into SQL Server at a deep level and really buffed up my development side. I started really getting into ALM in 2003; I’d been following Brian Harry for years, even before he joined us at Microsoft, and met with him in Raleigh when it was just 3 people sitting down and talking about what was going to become TFS. I knew ALM was really going to take off with Team System, which was what it was called at this point. I was the first guy who even knew what ALM was when I joined my current team – now we have a whole stable of people, we can’t hire them fast enough it seems.
What I Learned from the Military: Before that though I served in the US Army for about 12 years, starting in 1974. I’d just turned 17, and five days after my birthday I went into the service. Being a good soldier really comes down to the same qualities it takes to be a good IT person. Just for example, you must have absolute honesty and open communication because lives depend on it. So, I didn’t mind a subordinate telling me I was a butthole – hey, maybe he’s telling me something I need to hear!
The second thing had to do with that now cliched saying, “Be All You Can Be”. There really was nothing to prevent me from being anything I wanted to be, anything the team needed me to be. The military taught me that growth mindset. You really can do anything you want to do – they will pave the way for you to do that.
They instill cross training as a discipline. I went from being an infantryman to a medic, and then to communications. That’s learning, and those are the challenges that really gets me up in the morning and excited. And it’s great for the team. I never could say to someone on my squad, you’re light weapons not heavy weapons, so I can’t talk to you about radios and communications. Well, what if you lose someone on the team? Somebody needs to take over that slot.
In one training exercise, I was leading the team when we were trying to take a hill – the senior leaders then promptly had me “killed”. Now what? That really forces the team to improvise and rely on others. The self confidence that gives you to find all these abilities you didn’t know you had – well, you don’t often get that in the private sector.
Common Failure Points: The failure points in DevOps that I see, they’re exactly the same ones I see with Agile adoption. One of the most common is the lack of planning. One company I’m working with currently is just getting started. But I can already tell they’re going to be successful; their executives are committed, they’ve built a holistic plan much like we did at Microsoft, and they’re really committed to growing new skillsets and education. They’re going to win because the top leadership is sending a consistent message; other places, the executives are giving it lip service and it’s really a grass roots effort; those have a very high failure rate.
A few months back I visited another company that’s almost ten years along in their DevOps effort. My contact is all in, and so is his manager – but that’s as far as it goes. The pushback and friction he faces is enormous. I go onsite, share our success stories and those of other companies, and then look at what they’re doing and provide feedback. And we’ve had to be very frank – unless you get executive sponsorship you are going to be spinning your wheels, fighting a lot of resistance to change, which in turn will create waste. Ten years along, and they are still stuck in this very fixed mindset.
That’s two companies with very different trajectories, all because of executive buy-in.
Getting Executive Buy-In: To overcome this, I engage with the executives on their level and speak their language – not that of an IT person or a developer. I must use the terms that are key to them. Delivering business value. Delivering more value that causes the customers to buy more, to have them upgrade. To reduce the impact of defects. Showing them how the cost of fixing things early vs after shipping.
We’ve had decades of studies showing the cost differences in catching defects early in software development and how it equates directly to money. In a way this is a lot like when we work with executives on the adoption of test driven development. It’s the same problem really: why are we doing something that looks like it will cost me more money up front?
You need to present this to them as, you are going to make more money and save more money in the end. You’re going to have greater customer loyalty, increased sales, reduce the cost of your bottom lines, drive up your stock price – that’s what they want and need to hear.
The number one point that I want to bring out is waste. Executives get this instinctively once you connect your deployment practices with waste. Here’s the waste, and if you eliminate this, look as the payoff – that gets you the buy in you needed.
Getting To Lead Time: When we’re engaging with a customer we always start with a value stream analysis. It’s really vital to get to the lead time – once a customer gives me a request, how long does it take me to get to it – and cycle time, how long does it take to get that feature into production. Those are always the two metrics we’re trying to improve and we keep it top of mind.
But from there it becomes very client specific. For a team that’s been doing things in Agile for a while, our approach is going to be very different than for a team that is still stuck in waterfall. So we do a two part assessment; one of their ALM processes and Agile/DevOps maturity, and a second one a more client-specific risk analysis. From that assessment we start to build out a plan – what we will focus on, what the milestones will look like, and who owns things.
That last point is really key. I want a customer name attached to almost anything; that’s a way bigger struggle than getting a company to adopt moving to the cloud. But it’s absolutely vital – without clear ownership, the plan isn’t going to move forward.
A Fresh Start: Our basic approach with these DevOps roadmaps is to handle things like we do with Agile – we start small, hopefully with a new team and a greenfield project – we get our iterations going, and refactor as we go along. The basic idea is to find a good candidate for a fresh start, you get some traction and momentum built up – and out of this team you assemble some champions, people indoctrinated and totally bought into DevOps as a process.
To change the culture of a large org and get on a DevOps journey, you need champions – then you take those people and infiltrate other teams with them, and spread that success, that new way of doing things, elsewhere.
Plan the Work, Work the Plan: My military background taught me to put a lot of work into building a plan – that’s our starting point – and then as things progress we change the plan and adapt. That’s key with these roadmaps – to have the right people, committed, and to have key milestones. When we do our assessments one of the first things that pops up is the people part of things – does everyone have what they need as far of training? The answer is usually no, and it’s a lot of work to get everyone on the same page. That could be a milestone by itself – “By the end of April, everyone will have had an opportunity to go through training.”
The key point here in planning is not to get bogged down. We’re trying to turn the lights on. The more we turn the lights on, the less resistance we are going to face. By this I mean – transparency, dashboarding, and a key understanding and agreement on where we are, where we want to go, and how we’re going to get there. Everyone needs to understand what is the plan and the change that is going to happen, and what is expected from them.
Inevitably when people hear “plan” they think, “oh you mean project planning.” And that’s totally not the case! The plan can be verbal, not even written down – but you must have a plan to get there. And it can’t be – “let’s do DevOps – everybody get there on their own.” You have a plan when you’re going to work on your car right? Or work with your children. Is it written? No, but you do have a plan. This is not project management. But it–is a plan – based on gaps you found in your assessments. Your deliverables could be, teams formed, or everyone trained by X date.
Testing: Test automation is another common obstacle for most organizations. My first 10 years at Microsoft I would preach until I was blue in the face that you should automate everything, which is possible with a three-year release cycle. When Agile came around all that got thrown out the window, and testers got left in the dust. And now with DevOps – you flatten things out, where everyone has the same title – software design engineer. Everyone on the team is a developer regardless of what you do. And testing is now part of the job description of every person on the team.
Nowadays, I no longer say, “You MUST automate EVERYTHING!” – it’s just not possible anymore in most cases. You have to figure out where you’re going to get the most value from testing and focus on that. With TDD, you’re working on test layer while the development is happening because both those roles are embedded with the team – DevOps and TDD go together like peanut butter and jelly. The cycle is so short now that testing the UI layer and lots of brittle acceptance tests makes almost no sense.
That shift has been a huge challenge for most organizations I visit. We still see siloes where testing is kept separate and code is thrown over the wall – even if the developer and the tester are one cubicle apart. I think there’s one team I worked with in Portland that had integrated testing teams with the developers – their code coverage metric was 100%, and they always kept it at that level.
But that’s just one company out of the hundreds of companies that I’ve worked with! Every other customer I’ve worked with, they box off the testers and keep them separate. And it just doesn’t work… Let the deep testers do what they do well – higher level integration level testing, and testing in production. For example, with the VSTS program team, with feature flags, we ship with it turned off, then turn on the next sprint, using rings – internal first, then select customers, etc.
Architectural Decision Points: I don’t feel like “if you’re doing DevOps right you must go to the cloud.” I can implement the same thing in my org without the cloud and still be successful. I also don’t feel like you need to adopt microservices to be successful. Bold statement here – I don’t think DevOps has anything to do with architecture. DevOps can be successful regardless of how you are building your solution.
Just for example, the UI for Visual Studio isn’t built as a microservice. But when we build new feature sets, a lot of them are built as microservices for scalability. Microservices are terrific as a model, and it’s no different than what we were taught in college when it comes to programming – don’t make this behemoth method – make a method that is much smaller, more easily traceable within the code, and more easily changeable. Microservices gives us that and really improves scalability.
The same thing is true with containers, like Docker and Kubernetes. These technologies are great if you can do it as an architectural change – I believe in those solutions, they’re fantastic, but there’s no way I’m going to take something and rearchitect it just because it’s a better solution. Often, we have to drill down and ask – is this a business solution or is it an engineering exercise? Which means is it making us money, or saving us money? If not, whether it’s “better” or not, you should drop it.
Feedback and Learning: We talked about flow – another key piece is getting feedback. Most companies simply don’t think to implement things to see that they’re getting consistent, frequent feedback from customers. In your customer facing apps, do they have a way of providing feedback – a smiley face, or a direct way to call on a new feature? I don’t see orgs doing enough around the feedback loop; if they do have a feedback loop its manual, with long delays and full of waste and misinterpretation.
Another short-circuit to the learning process some companies fall into is with analytics. I’m still seeing a lot of people rolling their own analytics. That’s ridiculous – they really need to be using a vendor! I don’t care who you use, dammit, just use pick one. Trust with one vendor is a key aspect of Deming’s teachings. You really can’t put enough stress on how important it is to get feedback, more often, more accurately to the people that need it. In Lean Manufacturing this isn’t as big of a deal – you can’t change a car platform every month – but in software you can and should be adjusting and changing almost by the day.
The Roots of DevOps: DevOps is a really powerful movement and people often don’t realize how long its roots are. Twenty years ago, we had something called “MSF Mindsets” that had some very simple principles:
- Foster a team of peers
- Focus on business value
- Keep a solution perspective
- Take pride in workmanship
- Learn continuously
- Internalize qualities of service
- Practice good citizenship
- Deliver on your commitments
You know, I still go back to that list of guidelines when I get stuck. In a lot of ways, what we see today in DevOps inherits quite a bit from these simple daily ways of working with others.