DevOps Fragility, Antipatterns, and Consequences
Mark Eisenberg and Keith Anderson spotlight app architecture, organizational structure, and testing responsibilities and the importance of how these areas can improve or hinder the effectiveness of DevOps practices.
DevOps was born from the idea of applying development practices and tooling to the creation and management of IT infrastructure. The specific practice that caught the attention of these IT experts was Agile. They saw the promise of applying iterative development to infrastructure automation in a practice that would come to be known as infrastructure as code.
From that simple idea, DevOps has expanded to a set of practices that are intended to improve the speed of the delivery of value while improving the quality of the software deliverables. DevOps requires a full analysis of the value chain connecting the needs of the business stakeholders to the needs of the operations stakeholders. That analysis yields many blockers to delivering code quickly that does what it is supposed to do. The three we will be discussing here are application architecture, organizational structure, and testing.
Team Composition (creating a ‘Dev Ops’ Team)
One tenet of DevOps philosophy is to give the entire developer team the shared responsibility for the application through all phases of the application lifecycle, including deployment and operations. This shared responsibility, sometimes called shared pain, is designed to mitigate the issue of a team taking care of one phase, then washing their hands of it, throwing the next task over the wall to the next team. All too often this resulted in confusion, mismatched expectations, build failures or deployment failures that broke the application.
However, repeatedly, enterprises create a ‘dev ops’ team outside of the team of developers. This ‘dev ops’ team is usually responsible for creating pipelines, advising on source code branching strategies, infrastructure, security, and any number of other concerns that used to be under the umbrella of ‘operations’. This completely misses the purpose of Dev Ops philosophy and does nothing to address the original problem. Proper team composition would see team members who were good at deployment pipelines and operations sit side by side with team members who were good at test scripting and writing code. They would all be on the hook for a proper CI/CD (Continuous Integration and Continuous Delivery) pipeline, properly exercised tests, and deploying code that worked as intended.
The same anti-pattern is also common with security and quality assurance. Many enterprises understand that by keeping teams dedicated to these aspects of the development lifecycle outside of the development team, they are breaking the Dev Ops methodology, but they seem uncertain how to fix it.
Teams of software engineers can happily plan sprints, hold daily scrums, estimate, own and deliver work items from a back log, but without proper team composition, there will always be friction that results in anger, confusion, and dysfunction.
Lack of shift-left testing
Prior to DevOps and Agile, the burden of testing an application usually fell to a team specially focused on exercising test cases during a specific phase in the development lifecycle. One tenet of DevOps is about making all the individual loops within the dev lifecycle continuous, in that they are happening constantly during the cycle. Testing is one of the more difficult things to reorganize into a DevOps continuous pattern.
The solution is simple enough to understand but challenging for many teams to implement correctly. Unit tests allow testing to ‘shift-left’ and introduce quality control as soon as the first line of code for a new feature is written. In practice, many organizations do not mandate unit tests or with enough coverage to be effective.
By not implementing unit tests correctly or completely, future changes will eventually introduce regression bugs that never get uncovered prior to a release and can be insidious to find.
There is also a team composition aspect to this. Many organizations maintain separate test teams rather than bring test engineers onto the dev team, making the same mistake as forming a ‘DevOps’ team discussed above, resulting in the same friction and lack of accountability.
One of the tenets of DevOps is the need to be able to deploy new functionality to production without breaking existing functionality. The best way to do this is for the application architecture to be granular enough to permit portions of the code to be deployed without impacting all the code. To be truly independently deployable, the software components must also be loosely coupled to each other. The interfaces of the components are opaque. The internal state and function of a component must not be visible to any other component. Only the interface is visible. In this way each component can reach production while not impacting any existing component. The importance of this architectural approach cannot be understated when seeking to achieve the full potential of DevOps practices. It is possible to reach the quality goals of DevOps without this approach, but the velocity component will be severely constrained without it.
We have chosen to highlight these three areas of DevOps implementation because they are critical success factors and without their inclusion in enterprise DevOps implementations there will be a significant gap between the goals and the results of the effort. Many great promises are made about what DevOps can deliver to an enterprise IT organization, but without a commitment to these key concepts, expectations should be moderated for the operational benefits that will be achieved. A careful reading of enterprise DevOps case studies will reveal not only the magnitude of the outcomes, but also the magnitude of the effort.