{"id":27375,"date":"2018-09-10T17:47:00","date_gmt":"2018-09-10T17:47:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/premier_developer\/?p=27375"},"modified":"2019-02-14T20:17:51","modified_gmt":"2019-02-15T03:17:51","slug":"devops-stories-an-interview-with-brian-blackman-of-microsoft-premier","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/premier-developer\/devops-stories-an-interview-with-brian-blackman-of-microsoft-premier\/","title":{"rendered":"DevOps Stories – An Interview with Brian Blackman of Microsoft Premier"},"content":{"rendered":"

App Dev Manager Dave Harrison<\/a> talks with Brian Blackman<\/a>, a Senior Consultant for Microsoft Premier. As an Azure development and DevOps consultant, he works with Independent Software Vendors (ISV), Enterprises, and Partners where his specialty is DevOps. Brian\u2019s focus is assisting customers in establishing a vision for DevOps, assessing where they currently stand, and then working with them to develop a roadmap to attain their vision. MCTP, MCSD-C++, CSM, CSP, SAFe Agilist.<\/p>\n

Note – these and other interviews and case studies will form the backbone of our upcoming book \u201cAchieving DevOps\u201d from Apress, due out in late 2018. Please contact me<\/a> if you\u2019d like an advance copy!<\/p>\n

\n

\"brianblackman\"<\/a>My story \u2013 well, I came from the OEM division, that\u2019s really hands on \u2013 including working with Windows Mobile, Windows for Automotive as a program manager. It was great stuff; very close to the metal, working right with automotive systems from a hardware \/ software interface point of view. At Microsoft Consulting Services (MCS) I got into SQL Server at a deep level and really buffed up my development side. I started really getting into ALM in 2003; I\u2019d been following Brian Harry for years, even before he joined us at Microsoft, and met with him in Raleigh when it was just 3 people sitting down and talking about what was going to become TFS. I knew ALM was really going to take off with Team System, which was what it was called at this point. I was the first guy who even knew what ALM was when I joined my current team \u2013 now we have a whole stable of people, we can\u2019t hire them fast enough it seems.<\/p>\n

What I Learned from the Military<\/strong>: Before that though I served in the US Army for about 12 years, starting in 1974. I\u2019d just turned 17, and five days after my birthday I went into the service. Being a good soldier really comes down to the same qualities it takes to be a good IT person. Just for example, you must have absolute honesty and open communication because lives depend on it. So, I didn\u2019t mind a subordinate telling me I was a butthole \u2013 hey, maybe he\u2019s telling me something I need to hear!<\/p>\n

The second thing had to do with that now cliched saying, \u201cBe All You Can Be\u201d. There really was nothing to prevent me from being anything I wanted to be, anything the team needed me to be. The military taught me that growth mindset. You really can do anything you want to do \u2013 they will pave the way for you to do that.<\/p>\n

They instill cross training as a discipline. I went from being an infantryman to a medic, and then to communications. That\u2019s learning, and those are the challenges that really gets me up in the morning and excited. And it\u2019s great for the team. I never could say to someone on my squad, you\u2019re light weapons not heavy weapons, so I can\u2019t talk to you about radios and communications. Well, what if you lose someone on the team? Somebody needs to take over that slot.<\/p>\n

In one training exercise, I was leading the team when we were trying to take a hill \u2013 the senior leaders then promptly had me \u201ckilled\u201d. Now what? That really forces the team to improvise and rely on others. The self confidence that gives you to find all these abilities you didn\u2019t know you had \u2013 well, you don\u2019t often get that in the private sector.<\/p>\n

Common Failure Points:<\/strong> The failure points in DevOps that I see, they\u2019re exactly the same ones I see with Agile adoption. One of the most common is the lack of planning. One company I\u2019m working with currently is just getting started. But I can already tell they\u2019re going to be successful; their executives are committed, they\u2019ve built a holistic plan much like we did at Microsoft, and they\u2019re really committed to growing new skillsets and education. They\u2019re going to win because the top leadership is sending a consistent message; other places, the executives are giving it lip service and it\u2019s really a grass roots effort; those have a very high failure rate.<\/p>\n

A few months back I visited another company that\u2019s almost ten years along in their DevOps effort. My contact is all in, and so is his manager \u2013 but that\u2019s as far as it goes. The pushback and friction he faces is enormous. I go onsite, share our success stories and those of other companies, and then look at what they\u2019re doing and provide feedback. And we\u2019ve had to be very frank \u2013 unless you get executive sponsorship you are going to be spinning your wheels, fighting a lot of resistance to change, which in turn will create waste. Ten years along, and they are still stuck in this very fixed mindset.<\/p>\n

That\u2019s two companies with very different trajectories, all because of executive buy-in.<\/p>\n

Getting Executive Buy-In:<\/strong> To overcome this, I engage with the executives on their level and speak their language \u2013 not that of an IT person or a developer. I must use the terms that are key to them. Delivering business value. Delivering more value that causes the customers to buy more, to have them upgrade. To reduce the impact of defects. Showing them how the cost of fixing things early vs after shipping.<\/p>\n

We\u2019ve had decades of studies showing the cost differences in catching defects early in software development and how it equates directly to money. In a way this is a lot like when we work with executives on the adoption of test driven development. It’s the same problem really: why are we doing something that looks like it will cost me more money up front?<\/p>\n

You need to present this to them as, you are going to make more money and save more money in the end. You’re going to have greater customer loyalty, increased sales, reduce the cost of your bottom lines, drive up your stock price – that’s what they want and need to hear.<\/p>\n

The number one point that I want to bring out is waste. Executives get this instinctively once you connect your deployment practices with waste. Here’s the waste, and if you eliminate this, look as the payoff \u2013 that gets you the buy in you needed.<\/p>\n

\"roadmaplessons\"<\/a>Getting To Lead Time:<\/strong> When we\u2019re engaging with a customer we always start with a value stream analysis. It\u2019s really vital to get to the lead time \u2013 once a customer gives me a request, how long does it take me to get to it \u2013 and cycle time, how long does it take to get that feature into production. Those are always the two metrics we\u2019re trying to improve and we keep it top of mind.<\/p>\n

But from there it becomes very client specific. For a team that\u2019s been doing things in Agile for a while, our approach is going to be very different than for a team that is still stuck in waterfall.\u00a0 So we do a two part assessment; one of their ALM processes and Agile\/DevOps maturity, and a second one a more client-specific risk analysis. From that assessment we start to build out a plan \u2013 what we will focus on, what the milestones will look like, and who owns things.<\/p>\n

That last point is really key. I want a customer name attached to almost anything; that\u2019s a way bigger struggle than getting a company to adopt moving to the cloud. But it\u2019s absolutely vital – without clear ownership, the plan isn’t going to move forward.<\/p>\n

A Fresh Start:<\/strong> Our basic approach with these DevOps roadmaps is to handle things like we do with Agile \u2013 we start small, hopefully with a new team and a greenfield project \u2013 we get our iterations going, and refactor as we go along. The basic idea is to find a good candidate for a fresh start, you get some traction and momentum built up \u2013 and out of this team you assemble some champions, people indoctrinated and totally bought into DevOps as a process.<\/p>\n

To change the culture of a large org and get on a DevOps journey, you need champions \u2013 then you take those people and infiltrate other teams with them, and spread that success, that new way of doing things, elsewhere.<\/p>\n

Plan the Work, Work the Plan:<\/strong> My military background taught me to put a lot of work into building a plan \u2013 that\u2019s our starting point \u2013 and then as things progress we change the plan and adapt. That\u2019s key with these roadmaps \u2013 to have the right people, committed, and to have key milestones. When we do our assessments one of the first things that pops up is the people part of things \u2013 does everyone have what they need as far of training? The answer is usually no, and it\u2019s a lot of work to get everyone on the same page. That could be a milestone by itself \u2013 \u201cBy the end of April, everyone will have had an opportunity to go through training.”<\/p>\n

The key point here in planning is not to get bogged down. We’re trying to turn the lights on. The more we turn the lights on, the less resistance we are going to face. By this I mean \u2013 transparency, dashboarding, and a key understanding and agreement on where we are, where we want to go, and how we\u2019re going to get there. Everyone needs to understand what is the plan and the change that is going to happen, and what is expected from them.<\/p>\n

Inevitably when people hear \u201cplan\u201d they think, \u201coh you mean project planning.\u201d And that\u2019s totally not the case! The plan can be verbal, not even written down \u2013 but you must have a plan to get there. And it can\u2019t be \u2013 \u201clet\u2019s do DevOps \u2013 everybody get there on their own.\u201d You have a plan when you’re going to work on your car right? Or work with your children. Is it written? No, but you do have a plan. This is not project management. But it\u2013is a plan – based on gaps you found in your assessments. Your deliverables could be, teams formed, or everyone trained by X date.<\/p>\n

Testing:<\/strong> Test automation is another common obstacle for most organizations. My first 10 years at Microsoft I would preach until I was blue in the face that you should automate everything, which is possible with a three-year release cycle. When Agile came around all that got thrown out the window, and testers got left in the dust. And now with DevOps \u2013 you flatten things out, where everyone has the same title \u2013 software design engineer. Everyone on the team is a developer regardless of what you do. And testing is now part of the job description of every person on the team.<\/p>\n

Nowadays, I no longer say, \u201cYou MUST automate EVERYTHING!\u201d \u2013 it\u2019s just not possible anymore in most cases. You have to figure out where you\u2019re going to get the most value from testing and focus on that. With TDD, you\u2019re working on test layer while the development is happening because both those roles are embedded with the team \u2013 DevOps and TDD go together like peanut butter and jelly. The cycle is so short now that testing the UI layer and lots of brittle acceptance tests makes almost no sense.<\/p>\n

That shift has been a huge challenge for most organizations I visit. We still see siloes where testing is kept separate and code is thrown over the wall \u2013 even if the developer and the tester are one cubicle apart. I think there\u2019s one team I worked with in Portland that had integrated testing teams with the developers \u2013 their code coverage metric was 100%, and they always kept it at that level.<\/p>\n

But that’s just one company out of the hundreds of companies that I’ve worked with! Every other customer I\u2019ve worked with, they box off the testers and keep them separate. And it just doesn\u2019t work\u2026 Let the deep testers do what they do well – higher level integration level testing, and testing in production. For example, with the VSTS program team, with feature flags, we ship with it turned off, then turn on the next sprint, using rings – internal first, then select customers, etc.<\/p>\n

Architectural Decision Points:<\/strong> I don\u2019t feel like \u201cif you\u2019re doing DevOps right you must go to the cloud.\u201d I can implement the same thing in my org without the cloud and still be successful. I also don\u2019t feel like you need to adopt microservices to be successful. Bold statement here – I don’t think DevOps has anything to do with architecture. DevOps can be successful regardless of how you are building your solution.<\/p>\n

Just for example, the UI for Visual Studio isn\u2019t built as a microservice. But when we build new feature sets, a lot of them are built as microservices for scalability. Microservices are terrific as a model, and it\u2019s no different than what we were taught in college when it comes to programming – don’t make this behemoth method – make a method that is much smaller, more easily traceable within the code, and more easily changeable. Microservices gives us that and really improves scalability.<\/p>\n

The same thing is true with containers, like Docker and Kubernetes. These technologies are great if you can do it as an architectural change – I believe in those solutions, they\u2019re fantastic, but there\u2019s no way I’m going to take something and rearchitect it just because it\u2019s a better solution. Often, we have to drill down and ask \u2013 is this a business solution or is it an engineering exercise? Which means is it making us money, or saving us money? If not, whether it\u2019s \u201cbetter\u201d or not, you should drop it.<\/p>\n

Feedback and Learning:<\/strong> We talked about flow \u2013 another key piece is getting feedback. Most companies simply don\u2019t think to implement things to see that they’re getting consistent, frequent feedback from customers. In your customer facing apps, do they have a way of providing feedback \u2013 a smiley face, or a direct way to call on a new feature? I don’t see orgs doing enough around the feedback loop; if they do have a feedback loop its manual, with long delays and full of waste and misinterpretation.<\/p>\n

Another short-circuit to the learning process some companies fall into is with analytics. I’m still seeing a lot of people rolling their own analytics. That\u2019s ridiculous \u2013 they really need to be using a vendor! I don\u2019t care who you use, dammit, just use pick one. Trust with one vendor is a key aspect of Deming\u2019s teachings. You really can\u2019t put enough stress on how important it is to get feedback, more often, more accurately to the people that need it. In Lean Manufacturing this isn’t as big of a deal – you can’t change a car platform every month – but in software you can and should be adjusting and changing almost by the day.<\/p>\n

The Roots of DevOps:<\/strong> DevOps is a really powerful movement and people often don\u2019t realize how long its roots are. Twenty years ago, we had something called \u201cMSF Mindsets\u201d that had some very simple principles:<\/p>\n