DevOps Dojo – OKRs (Objectives and Key Results)

DevOps Dojo

1. What is vision?
2. What is mission?
3. What is strategy?
4. What are OKRs?
5. What is the relationship between vision, mission, strategy, and OKRs?
6. What are good OKRs and troubled OKRs?
7. What are the benefits of OKRs?
8. Where to deploy OKRs? At which level of the organization?
9. How do we score OKRs?
10. How often should we run an OKR cadence?
11. What is the process of setting OKRs?
12. What is the relationship between OKRs and Agile?
13. What is the difference and relationship between OKRs and KPIs?
14. Should we use OKRs to measure performance?
15. What tools are available for us to use to manage OKRs?
16. How does culture impact OKRs?
17. What are the pitfalls to watch for when deploying OKRs?
18. How do Microsoft Product Groups such as Azure DevOps or GitHub use OKRs?
19. How do we effectively write OKRs and are there any tips?
20. Any recommendations for OKR readings?

In the Dojo White Belt Master Class, the first session is OKR fundamentals. In this session, we answer these top 20 questions related to OKRs, then we ask participants to set their team OKRs for the rest of the class. Every day, each team checks their progress towards their committed OKRs and aspirational OKRs. On the final day, each team reports back with their overall OKR status and the outcomes of their learning experience.

This blog is not an introduction to OKRs. You can probably find many books, YouTube videos, and other free resources to learn about OKRs (e.g. Measure business outcomes by using objectives and key results – Cloud Adoption Framework | Microsoft Docs). This blog is about the challenges and our learnings of adopting OKRs in the Dojo community. In this community, members come from all walks of life in Microsoft and from various regions and business units to contribute to the Dojo InnerSource using their spare time outside of their day-to-day work. As you can imagine, the challenges and opportunities can be very different from the well-established organizational OKRs backed up by stable resources and funding.

In this blog, we will explore the following four questions using our personal experiences to demonstrate the many shades of gray in the Dojo community’s adoption of OKRs.

1. How do we develop an OKR that delivers outcome instead of output in the Dojo community?
2. How do we develop relatively stable OKRs with predictability in the Dojo community?
3. How do we measure our OKRs that are focusing on cultural aspects in the Dojo Community?
4. Can a community-based crowdsourcing/freelancing model work at the enterprise scale to deliver OKRs?

Question #1: How do we develop an OKR that delivers outcome instead of output in the Dojo community?

Microsoft’s FY22 ends on June 30th, 2022, and we were informed by GitHub in March that the Dojo community can take multiple newly released GitHub certifications for free with amazing prizes by June 30th. Since we release OKRs quarterly in Dojo community, it was perfect timing to add an OKR related to this in March for our last quarter of FY22.

The Dojo community understands the challenges of DevOps and what GitHub can bring, which includes Developer Velocity, Productivity, Collaboration, InnerSource, Innovation, and much more. We have done many internal and external GitHub related events with tremendous success. Team members recently suggested making an OKR to achieve a 95% and above passing rate for at least one GitHub certification by Q4 of FY22.

On the surface, it made sense because this was the best opportunity to enroll more people in the Dojo core team to become competent in GitHub technology since we are running front and center on promoting GitHub. On top of that, who wouldn’t like the amazing prizes and free registration?

But when we took a deeper look and asked ourselves a few critical questions, we started to debate if this was indeed a good OKR. Here are some questions from the team:

1. First, the Dojo community is a highly diverse community with people from various expertise areas in project/product management, process, and technology, so not everyone is technical and making non-technical people take the technical certification does not match their intrinsic drive. OKRs can be only achieved by those who are committed to achieving it or at least aspire to achieve it. The assumption that everyone would be interested in taking a technical certification has flaws in this case, so it will not be achievable even if the intention is great.

2. As mentioned, this quarter would be our last quarter in FY22, and the last quarter has always been the busiest time of the year. This puts further time constraints on anyone who needs to take extra work to prepare for a certification, so this constraint makes the OKR less likely to be achieved.

3. Another important aspect of OKRs is about outcome, not output. Having 95% and above passing rate of certification is an output, but not yet an outcome.

So, what is the outcome?

“An outcome is a change in human behavior that drives business results.”

– Josh Seiden, Outcomes Over Output

Most of us don’t have clarity regarding the difference between output, outcome, and impact. Specifically, we have the most ambiguity on outcome. People on the ground create features, but creating features is creating output that may or may not lead to an outcome. Executives care about impact which includes financial benefits such as lower cost, higher revenue, better profit, and greater growth. People seem to have problems linking the output on the ground to the impact at the top.

Let’s come back to our scenario. Completing a certification doesn’t mean you have a change in human behavior that drives business results, so this is an output. We can achieve an outcome if we change the OKR to be:

“Have the ability to coach GitHub Labs in our Internal Dojo Master Class with a final satisfaction rating above 4.0 out of 5.0”.

In this case, there is a change in behavior, because you take the learning as a learner after you pass the certification to apply your learning in teaching as a coach. In our experiential learning blog, we have demonstrated that teaching is the best approach in all learning methods. You might not realize it, but you are doing Dojo’s SHU and HA in this scenario.

Now, having an opportunity to coach in the internal Master Class doesn’t mean that you have achieved the business results. Having an overall satisfaction rating from your class would have a positive correlation with your outcome.

A better OKR would be:

“Anyone who is committed to taking the GitHub Certification can deliver GitHub Labs in our internal Dojo Master class with a final satisfaction rating above 4.0 out of 5.0 by June 30, 2020.”

However, there is one problem in this KR. In three months, not only do you have to pass the certification, but you must also have an opportunity to coach the internal audiences in one of the master classes. However, we might not have internal master classes available in their region in the last quarter, so your output (passing the certification) cannot be proven to have become an outcome (behavior change from learner to trainer). Therefore, the time in KR is not feasible.

As you can see, defining a good OKR is hard as it takes a lot of deep thinking, questioning, refining, and often debating to come up with.

Question #2: How do we develop relatively stable OKRs with certain predictability in the Dojo community?

If you use OKRs to measure your outcome, you probably use this grading system. Even though there are some regional differences, in general this is the norm. Most likely you will have some red at the beginning as it can take two to three quarters to get it right. You eventually want to see mostly green (0.7) and some yellow. All dark green OKRs show that you might not have stretched yourself enough, while too much red shows that you didn’t plan well, which is discouraging to the team. If you are colorblind, please use arrow-keys to translate the meaning of the colors.

Here are our OKR scores for the past three years.

The number indicates the KR, for example, #1, #2, #3, #4 in Q1 Year 1 means we had 4 KRs in Q1. The color yellow shows solid progress, green shows you nailed it, and red shows no progress or funding.

The observation:

1. We had red in all four quarters in the first year. It really took us a year to create more predictable and stable OKRs.
2. There is a missing quarter due to a large internal reorganization. As much as the community is organization agnostic, sometimes it does impact the community as shown in this case.
3. Over time, we have become better at defining stable OKRs and the number of OKRs has increased from 4 to 5 or 6, so the velocity increases as we get better at OKRs.

Challenges in defining OKRs in a community:

1. People come and go, all work is done by volunteers, and there is no mandatory and forced work by the community, so it is highly variable and unpredictable in nature.
2. Even if members are motivated to do the work, their daily work schedule and personal situation can change very quickly from week to week or month to month.
3. People change jobs, are promoted to different roles, or move on to something else so they lose their interest or drive to continue doing community work.
4. All those aspects make defining OKRs challenging.

The key elements of having predictable OKRs in community:

1. The community work needs to be interesting to the members who want to learn and master something that is outside of their day-to-day work.
2. The community work needs to be done in a small group (4–-6 people) who trust each other and like to work together as a team.
3. A lot of pre-work must be done and vetted before making a public OKR. Sometimes it is a judgement call on the new initiatives, but still, pre-planning to understand the risk and dependency is absolutely required to have a good OKR.
4. We do add last-minute OKRs due to immediately available funding, which is always gold.
5. We also remove OKRs during the first month of the quarter if the dependency can’t be resolved.
6. All those aspects make defining OKRs more stable.

Question #3: How do we measure our OKRs that are focusing on cultural aspects in the Dojo Community?

Dojo’s vision: Spreading the Power of DevOps Culture.

Accepting the fact that DevOps culture is the most important part of DevOps is only the first step. The million-dollar question after that is how do we measure DevOps culture in OKRs?

As the old saying goes, “Not everything that counts can be counted. Not everything that can be counted counts.” Culture is one of few things that is so hard to count but absolutely counts.

In life or at work, there are a few examples that are similar to culture.

One example is at the enterprise level, where we know that profits and revenue are easy to measure, while delighting customers is a lot harder to measure but matters the most.

Another example is at the personal level, where many people chase wealth, status, and achievement because progress is easy to measure, while failing to realize that the gains that count the most are the hardest to count. Real growth is building character—striving to improve in generosity, integrity, humility, fairness, or courage.

So how do we measure culture and mindset improvement in the Dojo community? It is quite hard, but we figured out a few ways that are direct and indirect indicators of culture and mindset improvement.

Metric #1: The number of initiatives we deliver each year.

This might be controversial because the number of initiatives we deliver sounds like an output, not an outcome. But in a community setting, this has very different meanings and consequences.

All initiatives are done by self-forming teams where each team can be 4–10 people who work together for 3–6 months. Not all initiatives are completed, but most of them are. Since this is all community work, it is beyond their day-to-day job. If they can stay together for 3-6 months or longer to achieve a goal, it is a great accomplishment. This is an indirect indicator of cultural change through cross-organizational collaborations.

In 2021, we completed a total of 41 initiatives. One key member of the Dojo community said, “We were not a team, we were friends, we were family working side by side supporting each other.” This is a behavior change to create business results.

There could not be a better way to end 2021 than with an Azure blog about DevOps culture and mindset. In the Dojo community, we focus on a culture of diversity, culture of experimentation, culture of team of teams, culture of psychological safety, culture of empowerment, culture of team recognition, and culture of being human-centric. Within a year, not only did we archive all our OKRs, but more importantly, we built an action-oriented, giving, sharing, learning, and loving community where we support each other in good times and in bad.

Metric #2: DevOps Dojo InnerSource Telemetry Dashboard

We list a few examples of the real-time insights from the Dojo InnerSource site. Through the data insights, we can reward the top contributors and the authors whose articles have the most views. We also understand the regional differences so we can do more internal awareness marketing and enablement.

1. Views by country
2. Top 10 contributors by PR
3. Top 10 consumed articles

Metric #3: The number of people who are promoted and rewarded in Dojo Community

Microsoft has incredibly competitive and competent people, so getting promoted or rewarded is also highly competitive too. But many people, especially the core Dojo members, have been promoted and rewarded at an amazing rate that most of us have not seen in Microsoft. You might wonder why? Since the Dojo community is a community of doers, helpers, givers, and collaborators, we produce outcomes and generate impact, but more importantly, we build a loving community so each one of us has a huge support system beyond our regular organizational boundaries. In Microsoft, our performance is measured on three things:

1. Your key individual accomplishments that contribute to team, business, or customer results.
2. Your contributions to the success of others.
3. Your results that build on the work, ideas, or effort of others.

As you can see, the Dojo community provides great opportunities to support all three aspects of performance measurement, which is why the Dojo members have a higher chance to be promoted and rewarded.

Metric #4: The impacts of the events we run and feedback we received from culture/mindset change aspect.

Microsoft internal audiences are hard to please and they have limited attention due to busy schedules to support customers. The wisdom of the crowd really impressed our highly opinionated Microsoft colleagues, but more importantly, the speakers and the crew members from the community learned so much through the process and we have built long-lasting friendships from cross-organizational collaborations. A real-Life example of Outcomes over Output can be found in the impact created by the Dojo community and the feedback received as shown below.

“Really amazing and valuable lessons!”

“Great foundation on why GitHub!”

“Very clear and thorough explanation of GitHub’s features and roadmap.”

“Phenomenal series. Thank you for putting this together. I learned a ton!”

“What’s not to like? The session was informative and there was a fun contest!”

“I found the knowledge sharing very effective and easy way for me to remain updated.”

“These sessions are so well put together. The slides and demos are wonderful. Kahoot quizzes! 💪”

“Excellent Presentation. Can you please share the slides with us? I need to present the same to a customer.”

“Kudos for organising an awesome series of SHOTs sessions. Each of the sessions is very well crafted, capturing the details and links being shared in chat.”

“Thank you for hosting the DevOps SHOTS community. The latest series on GitHub was incredible with fantastic speakers! Key customer questions have been tackled in these sessions including how we use GitHub, what workflows are available, and best practices to follow.”

Question #4: Can a community-based crowdsourcing/freelancing model work at the enterprise scale to deliver OKRs?

In recent discussions with a few crowdsourcing/freelancing founders in startup companies, we have learned that they are looking to deploy this model into enterprise through our community as they see great potential. So, what is the rationale?

Before the pandemic, CEOs rated skill gaps and talent shortages as a top business challenge, and enterprises couldn’t hire all the talent they needed for their digital transformation. However, the COVID-19 crisis will not change this, and it will in fact make the situation worse due to the shortage of talent coupled with higher demand. With the pandemic, digital transformation is not just nice to have anymore, it is a must-have for survival.

We only have one earth to share and with much better awareness of sustainability in everything we do, many companies started to realize the benefits of sharing in enterprise. Sharing in business provides numerous social, economic, and environmental benefits to society. The sharing economy is growing, and it has proved to be a tremendous success in business and in society.

The community-based crowdsourcing/freelancing model creates a competitive advantage in this disruptive market by tapping into the undiscovered and trapped value in our organizations.

So how does it work in the Dojo community? We have tested this model for almost three years with a very consistent OKR performance that applies to both internal master classes and customer experiential learning classes. All classes received a rating of 4.0 and above out of 5.0, and in fact, most classes are 4.5 and above. So, what magic can achieve the OKRs?

First, DevOps is a very complex topic. In this case, we have 14 areas to cover in each master class—the 4 pillars, 8 capabilities, OKRs, and our own transformational story. In these 14 areas, we cover point of view, role play, hands-on labs, and a final challenge. It is impossible to apply the normal protocol to coach a room of up to 30 participants with 2–4 coaches while still delivering an immersive experience. With the pandemic, everything shifted to remote, creating a great opportunity for us to pivot to the time-sharing model. As you can see in this picture, we can have 10 to 14 coaches to cover various topics that they are experts in.

The benefits for us:

1. Preparing for such a class is a great internal collaboration exercise that creates a team-selecting team. We normally have two captains who recruit coaches. The captains gain tremendous leadership skills to orchestrate the event. This is aligned with Microsoft’s “Growth Mindset” which is a key attribute of Microsoft’s culture.
2. It is less frightening to everyone who delivers this class even though each person has accountability for their subject. With the support system in place, we feel great comfort that in case anything happens, you have backup and a team behind you.
3. Team building and people bonding between the coaches are a natural outcome of the intense and always dynamic exercise that each delivery is about.

The benefits for customers:

1. When we have over 200 years of diverse experience from the crew, our customers learned so much more than from a regular class of 2–4 coaches. In a remote session, it is much more interesting to hear different voices and different styles than the same voices and styles for a long period of time.
2. They watch how we operate as a single team because community members work together for a long time. We are well oiled machines; it shows in our delivery. As a result, it brings positive influence on the customers who come from different organizations, different skills, and different backgrounds. People learn more from how we act, not from how we talk.

Final Thoughts

Setting goals as outcomes sounds simple, but it can be hard to do in practice as you have seen in this blog. What makes it hard is that we often set goals as output or impact. When we set our goals as output, it is just a number, and it doesn’t mean it has value to our internal or external customers. When we set our goals as impact, it is too high level. For example, we ask teams to show business values (profitability, revenue, or cloud consumption), but these impact-level goals are too complex to be useful for our teams on the ground.

So next time when you are setting your outcome in your OKRs, ask these questions:

1. What are the internal and external customer behaviors that drive business results?

2. How can we get people to show more of those behaviors?

3. How do we know that our hypothesis on the outcome is right?

Adopting OKRs by a community is even more challenging than in a hierarchical organization as this blog demonstrates. However, if we ask our teams to work on an outcome, the smaller, more manageable targets, taken together, will create the impact we want. We do this by asking the team to focus on changing behavior in a way that drives culture and mindset change.

OKRs are both a cultural and process change. An important ingredient of success in adopting OKRs in any organization, groups, and communities is a mix of top-down and bottom-up involvement. It is really a way to build a social contract in your organization that says I’m going to sign up to do this amazing stuff. Having the opportunity to meaningfully contribute to what you will be held accountable for and what you are passionate about goes a long way in enhancing your work and engagement with others.

We would love to hear about your experience and learnings in adopting OKRs in your community. We believe sharing significantly improves sustainability on earth.

Special Dedication

This blog was written and reviewed by the following authors in the Dojo Community.

• April Edwards April.Edwards@microsoft.com
• Rui Melo Rui.Melo@microsoft.com
• Dave Burnison dabu@microsoft.com
• Gautam Patel gautampatel@microsoft.com
• Gary Johnson garyjohnson@microsoft.com
• Kan Tang Kan.Tang@microsoft.com

DevOps Dojo Dojo Community, 1Microsoft DevOps Dojo

Follow