Randy Pagels takes you on a DevOps journey to frame up the way an organization would foster a strong DevOps culture.

Contributions made to this blog post by Kyle Burns, Microsoft Cloud Solution Architect and Able Wang, former Principal Cloud Advocate and DevOps Lead at Microsoft, his contributions and skills will be forever appreciated.

During this blog series I will be taking you on a DevOps journey to frame up the way an organization would foster a strong DevOps culture. I’ve referenced this fantastic blog written by Niels (Nells) Talens on the “AgileThings” website. Source: AgileThings. This blog is built upon the article and visualization and will walk you through an organization’s DevOps journey. The primary benefit of DevOps is that it gives the organization the confidence to change at the speed of business.

Overview of DevOps

At Microsoft, DevOps means one thing. “It is the union of People, Process and Products to enable continuous delivery of Value to our end users”. -Donovan Brown, Partner PM Azure Incubation Team. On the people side, this means breaking down walls between DEV teams, QA teams and Ops teams. Instead of three separate teams, have one team with everyone focused on one goal, to continually deliver value to our end users. Teams have become far more cross functional and own the end-to-end customer experience. The developer’s role is to implement a user story, write full coverage unit tests for that user story and either work with QA to write UI automation or writing the UI automation themselves to test the User Story. This way, developers can iterate fast enough and catch regression bugs without the need for months of regression testing, that’s such a manual process.

We’re also working hand in hand with the Ops folks to figure out what is the best way to deploy our applications consistently and quickly. For processing, there needs to be a process in place which will let teams iterate fast enough, produce code of high enough quality, and a way to deliver the application to the end users quickly and consistently. Agile processes (done correctly) work great. And finally, for fantastic end-to-end products and tooling, the answer is Azure DevOps & GitHub, a better together story.

DevOps Nirvana, what does this mean? Webster defines it as a state of perfect happiness. Although a complete DevOps end-to-end setup might not be perfect, it can be extremely close. By adopting the principles of agile, automation, continuous integration and continuous delivery, you will reach your goal of a healthy DevOps culture. This blog series will provide some key concepts which demonstrate Microsoft’s vision of using DevOps, GitHub, and Azure to achieve true DevOps Nirvana. 😊

What problems does DevOps address?

• DevOps addresses the risk associated with changes.
  • This risk arises from missed requirements, new bugs, failed deployments, etc.
• DevOps focuses on streamlining deployments to minimize the effort required to deploy software.
  • As applications complexity increases, we see deployments measured not in man hours but in man days.
  • Deploy more frequently, with reduction of pain associated with performing each deployment.
  • This also helps to ensure consistency so that QA looks like Dev and Production looks like QA.
• DevOps minimizes hand-offs between teams.
  • This is not only through leveraging technology, but also by changing the cultures of the teams involved.

Benefits of DevOps

• Significantly higher throughput.
  • High functioning DevOps organizations have drastically higher throughput. In these organizations code makes it from Commit to Production 2,555 times faster.
• Significantly higher employee loyalty.
  • Employees of these organizations are 2.2 times as likely to recommend their current employer to a friend as a “great place to work”.
• Significant increase in quality.
  • Unplanned work is reduced by 22%, leading to a 29% increase in new work completed by their development and DevOps team.
• Significant reduction in Security Issue mitigation.
  • A 50% reduction in the time spent addressing security issues.

At its core, DevOps is about working to change how Dev, QA, and Ops work together. Ultimately tearing down walls and working together for a common purpose, thus continual delivery of value.

Overview of Application

In early 2016, Abel Wang and I met in-person and worked to create an app and demo that was simple enough to setup, yet powerful enough to demonstrate some core concepts of DevOps best practices on Azure. During this time, Mercury Health was born, named after Greek Mythology by Jeff Fattic. This app is basically a simple health tracking application. It tracks the exercises that you do and food that you eat throughout the day. Then based on the calories in and calories out, it calculates BMI and draws some cool graphs to let you know if you’re healthy or not. I re-wrote and enhanced the app using .Net 6 to utilize some key concepts and Azure services such as App Service Plans, App Service, Key Vault, SQL DB, Configuration Store, Feature Flags, Application Insights, Log Analytics Workspace, Playwright UI Automation, Availability Testing, API Management, and Azure Load Testing for performance and scalability testing. The source code repo of choice is GitHub that uses the built-in tools, DependaBot, CodeQL, Advanced Security, and GitHub Actions for CI/CD.

Mercury Health Home Page

App Core Components

Azure Key Vault

For many applications, just keeping the secrets out of source control isn’t enough. Azure Key Vault helps solve the secrets management problem. Azure Key Vault can be used to securely store and tightly control access to tokens, passwords, certificates, API keys, and other secrets, such as database connection strings.

System Assigned Managed Identity

Managed Identity is an enterprise application within Azure Active Directory and used by any Azure resource to authenticate to any service that supports Azure AD. This app uses Managed Identity for App Service, Azure Functions and API Management. The keys are generated at infrastructure deployment time and inserted into Key Vault and added to the application setting using a special @Microsoft.KeyVault reference syntax.

App Dynamic Configuration

Azure App Configuration is a managed service that helps you centralize an application and feature settings simply and securely. Dynamic configuration handles changes dynamically without causing an application to restart.

Feature Flag Toggles

Feature flags is a software engineering technique that turns select functionality on and off during runtime. Feature flags are used to control exposure and really nothing more than if statements in code. The important thing is all the code is built, tested, and deployed together all the time. This allows you to have code in production that isn’t turned on.

Application Health Checks

Developers can think ahead and save themselves and others from future frustration. Health Checks validates that the web server and application code run in a way that can later have automation put around it to prove “is the web service or database itself up?”. Regardless of the technology stack, start your projects with a health monitoring HTTP endpoint. Start real-time monitoring of your application with health probes for containers, physical servers, or other application dependencies.

The Mercury Health Dev Team

The mission of Mercury Health is to help improve longevity and quality of life through healthy activity. The Mercury Health development team has embraced a DevOps approach and is using continuous integration for builds and unit tests. If everything passes they use continuous deployment that deploys to various environments. They use UI Automation tests are run to ensure usability and key functionally works as designed. Lastly, Load and Performance tests are run from various locations around the world at the expected peak load to ensure there are no challenges around performance or stability when operating at full capacity.

Modern Application Development Pattern

The development team follows the modern application pattern for building and deploying code using resilient DevOps practices.

The DevOps Journey

As we learned from blog post “AgileThings” you can break the journey down into 8 components. For this blog post series, I plan on diving into core sections Agile Development, Agile Testing, CI, CD, Automated Testing, and DevOps Automation.

Agile Development

Building on top of the most basic of Agile principles and processes are Agile development-focused concepts. Ensuring builds are automated, pairing up developers, working off user stories, employing the usage of feature toggles and continuing to break down systems into the smallest and most manageable pieces possible are all characteristics of a healthy development team; and therefore, would support the continuation of a DevOps journey.

Agile Testing

In order to continue the DevOps journey past basic Agile practices and development, a healthy testing environment must exist. As most of you probably know, in Agile, each sprint or iteration should result in code that is completely shippable. This doesn’t mean the code must go to production and be enabled after each and every sprint, but the code should be in a state that would support such a scenario.

This step focuses on manual testing while automated testing will build upon this step next. To have completely shippable code, the definition of done cannot be anything close to the old saying “it works on my machine.” Rather, code must be completely tested, and the larger team must be confident in a production release or migration activity at the end of that sprint. Therefore, many behavior driven testing activities must be perfected within the realm of manual agile testing. Specification testing (aka black-box testing), scenario testing, and specification by example are just 3 examples of manual agile testing activities.

Automated Testing

Assuming development and manual testing activities are in place correctly and are operationalized, the next step in the journey involves maturing the testing approach and automating as much as possible. Development processes such as test-driven development should be explored and perhaps implemented. In TDD, developers first write tests (that fail) which define a feature, function, or enhancement. The developers then write the minimum amount of code to pass that test and then refactor as needed. This scenario is repeated, causing many tests to be written without having to go back and add them at the end. These tests can then be automated as part of the normal build processes.

Behavior driven development is derived from and is just an extension of TDD that shares the same benefits. Many companies use BDD approaches with a common “Given-When-Then” acceptance criteria syntax which then allows acceptance criteria to be automatically converted into automated tests.

Regardless of which development processes are in place, the focus on this step is about automating the testing process.

Unit Testing – “it works on my machine”

Why is Unit Testing so important? Even if you’ve never heard the question asked, you’ve probably been in a situation where your team ignored unit tests or treated them as an afterthought. They’re often the first thing to go when time runs short. When I ask that question of app dev teams, more times than not I hear “there’s not enough time”.

When the Mercury Health app is built, all unit tests are run. Unit testing is part of the solution, not proving that the app works, but it’s a detector of unintended change. Code coverage metrics should correlate directly with the level of confidence teams have that the changes made or valid. Keep in mind, it’s still important that functional testing works to ensure the app does what it’s supposed to do.

If you ever wonder, how do you maintain quality in a DevOps world, unit tests are the answer! I don’t think it’s possible to be successful practicing good DevOps, if you don’t implement good unit tests. So how do you maintain quality during the dev inner loop? You don’t have time to do full end to end functional testing at the end of your development efforts, this could take months to do. You need to push code out now, every two weeks, every week, maybe even multiple times a day and having a battery of good unit tests is how you maintain that quality.

Continuous Integration

Once testing has been largely automated, continuous integration can be achieved. Martin Fowler defines CI as “a development practice that requires developers to integrate code into a shared repository several times a day”. Software that runs only on a developer’s machine isn’t providing value to customers of the software.

Each check-in or commit needs to be verified by an automated build, allowing teams to detect problems early. By integrating regularly, you can detect errors quickly, and locate them more easily. I would like to expand on that definition to point out that merely building only ensures the integrated code compiles. Therefore, to produce a healthy DevOps culture, execution of automated tests that validate the integrated code is required. This is why we place CI as the step after automating the testing processes.

Unit Testing – “it works in shared dev environment”

Azure DevOps is used at Microsoft daily and consists of 133K+ code check-ins, 340K+ builds, and 157K+ releases per day. On the Azure DevOps team, they transitioned from long running functional tests and shifted to using unit tests exclusively that run with each pull request. For instance, every time a developer checks-in code, it goes through a pull request and an integration build is run to make sure all the code works together. It has to compile and it must run through all of the L0 and L1 tests.

ADO Unit Tests in the pipeline for past 14 days

How many tests do we run? Every developer check-in must pass all 91,581 unit tests/per PR and running all of them run takes about 20 minutes. That’s how we maintain quality at Microsoft with Azure DevOps. How important is unit testing? Absolutely vital!!!

Green means Green and Red means Red. Only all-green builds get to be released.

Conclusion

Having Agile principles and processes across development-focused concepts is a key success factor to a healthy DevOps culture. Once Agile processes are in place, a consistent testing environment must exist. Manual testing to ensure each scenario is supported and working. Unit Tests supporting critical steps to make sure the product is completely tested and shippable. Continuous Integration setup and automated for detecting errors quickly. Execution of automated tests that validate the integrated code is required. Every developer goes through a pull request, a build, and MUST pass every L0 & L1 tests by running 91K+/per PR successfully.

Stay tuned for Part 2 in this series where I’ll focus on the importance of having good Continuous Delivery, a natural extension of Continuous Integration. The goal of Continuous Delivery is to put business stakeholders or customers – NOT IT – in control of an ongoing cycle of software releases. Also included will be deep dive on using Infrastructure as Code to automate deployments for consistent environment setup. In code, you define the infrastructure that needs to be deployed and now it’s versioned as part of the project in the source code repo. Then, I’ll cover the importance of Automated UI Testing, Load and Performance Testing, and Application Monitoring. Finally, Part 3 in this series, I’ll walkthrough step-by-step how to setup and how to demo the Mercury Health app.

In the meantime, check out the GitHub Repo on Mercury Health.

As the Mercury Health team says, remember to think positively, stay focused, breathe, and be awesome!