Using monorepos to increase velocity during early stages of product development
This is a guest post by Victor Savkin, co-founder of Nrwl. Read more about Victor and his company at the end of this article.
Building a new product requires a lot of experimentation. Applications appear, disappear, merge, divide. Code has to be shared. Ownership changes. People move from team to team, and they have to wear different hats and contribute to many projects (backends and frontends) at the same time. CI (continuous integration) and deployment processes change weekly. These realities can introduce a lot of friction.
A Nx-based monorepo helps remove the friction, and create a setup where the boundaries can be redrawn until a good architecture emerges. In this article we will look at why this friction appears and how using a monorepo can mitigate it. We will also look at an example of a company that rapidly developed a new platform using this approach.
Public vs published
To understand the effects of monorepo-based development on velocity, we need to first understand the distinction between public and published APIs. Imagine we have a class, say a UI component. Its public API is a set of props (or inputs). Now let’s suppose we need to change one of the props. Maybe the schema on the backend changed, a field was renamed, and we need to reflect this change in the UI.
If we can change this UI component with all of its consumers in a single PR, and verify that the change is correct, this will only take a few hours of dev time. This isn’t a big deal.
Now, imagine several microfrontends, in separate git repos, using this component. If that’s the case, to make this change we will have to send a PR to the component’s repo, merge it, then release a new version of this component, and, finally, update all microfrontend repositories.
And only then will we know if the change actually works for all of them. If it doesn’t, we will have to make another change, release another version, and try again.
In other words, if the consumers of this component aren’t known, or known but hard to update, this change suddenly becomes an effort that can take weeks, and in some cases is not possible at all.
That’s the distinction between public and published APIs. Changing a public API requires us to update the clients of the API, but it’s possible to do it in a single go. In the case of a published API, it’s not possible to do it in a single go, and sometimes not possible to do at all. So changing a published API is a lot more effortful.
Observation: when something is effortful, developers avoid doing it. So developers avoid changing published APIs.
New systems vs old systems
Early in the development lifecycle, developers are still figuring out the system’s architecture and protocols, deciding on what can be shared, and so on.
As a result, the cost of introducing published APIs is a lot higher because the number of times we need to change it is a lot higher.
For instance, the following operations are very common in the first year of a system’s development:
Creating applications. By applications here we mean anything that can be run and deployed independently. Every microfrontend, every microservice is an application. It will happen multiple times a month.
Creating/extracting libraries. By libraries here we simply mean a piece of code with a well-defined public API. Libraries can be created to share code but also to organize code. It will happen every week.
Making code changes to shared libraries. It will happen every day. Refactoring multiple applications and libraries. It will happen every week.
When we develop every application and library in their own repository, we create published APIs. Using a monorepo helps us avoid that.
Now, with this in mind, let’s consider what developing in a monorepo can actually look like.
Monorepo-style development and Nx
Monorepo-style development is a software development approach where:
We develop multiple projects in the same repository.
The projects can depend on each other, so they can share code.
We can impose constraints on the dependency graph of the repo.
When we make a change, we do not rebuild or retest every project in the monorepo. Instead, we only rebuild and retest the projects that can be affected by our change.
The last point is crucial for two reasons:
It makes the CI faster. In some cases, overall CI time can be orders of magnitude faster.
It gives teams working in the monorepo independence. If two projects A and B do not depend on each other, they cannot affect each other — ever. Team A will be able to develop their project, test it, build it, merge PRs into the main branch without ever having to run any code written by Team B. Team B can have flaky tests, poorly typed code, broken code, broken tests. None of it matters to Team A.
Nx is a set of extensible dev tools for monorepos. It’s free and open-source. It’s built using TypeScript and Node.
It’s difficult to show the developer experience of Nx in writing, so instead, if you are interested in how-to, watch this walkthrough:
To give you an idea of what building a new large system in a monorepo looks like, let’s look at one of the companies that built a new platform in a monorepo.
The client team noticed their 10+ Angular applications had similar components and infrastructure. They realized that if they built a new platform, which would consolidate their applications and enable code sharing, they could increase their velocity and improve consistency across applications. They opted to use an Nx-based monorepo, and started carefully planning and designing.
Shortly after beginning the project, markets started to shift rapidly and the priorities changed. It became crucial that the new platform went to market as soon as possible. Doing careful and slow design was no longer something they could afford.
Applications started to onboard sooner than expected as rough UI designs were finished every week. The needs of these new applications required changing the core infrastructure and shared components.
GraphQL/NodeJS microservices were created to support the Angular applications. The microservices shared code with each other, and shared types with the frontends. The types were generated into Nx libraries using schema files.
Application designs were in constant flux and often reused pieces from other applications in development. Shared components were extracted from applications on a weekly basis, expanding and removing some of the components’ APIs.
Like components, the preemptive permissions, search, and notifications infrastructure required an overhaul.
The rate of change was high. And the number of changes going into the repo kept increasing as more applications were added, and more developers were working on them.
Even given all of this, a well-functioning monorepo allowed the teams to perform these changes at high velocity. They added, merged, and reshaped libraries. They performed large-scale refactorings of dozens of applications and libraries at a time. No matter how complicated and impactful the changes were, they could always verify that the changes are safe before merging them into the main branch. They changed the microservices’s APIs with their clients in a type safe fashion. All of these were done without blocking other developers contributing to the repo.
This was possible because they didn’t have any published APIs. No matter how many applications and libraries were developed, they could always change a piece of code with all its clients in a single PR. And they could verify that the change was safe before merging it. Many changes, which otherwise would have taken weeks, took only hours.
Concerns and misconceptions
There are some concerns and misconceptions about monorepo-style development. Let me address the most common ones.
Isn’t it a recipe for creating “a big ball of mud”?
This misconception comes from the fact that, in most repositories, any file can import any other file. Folks try to impose some structure during code reviews, but things do not stay well-defined for long, and the dependency graph gets muddled.
With Nx, we can create libraries that have well-defined, and enforced, public APIs. And because creating libraries takes just a few seconds, folks tend to create more libraries. So a typical application will be partitioned into dozens of libraries, which can depend on each other only through their public APIs.
Moreover, with Nx, we can add metadata to the libraries in our repo and define constraints based on it. For instance, we can statically guarantee that presentation components cannot depend on state management code.
Will it result in a monolith?
It’s a common misconception, which comes from a strong association of a repository with a deployment artifact.
But it is not hard to see that where we develop our code and what/when we deploy are actually orthogonal concerns. Google, for instance, has thousands of applications in its monorepo, but obviously, all of them are not released together.
So monorepo !== monolith. Quite the contrary, because monorepos simplify code sharing and cross-project refactorings, they significantly lower the cost of creating libs, microservices and microfrontends. So adopting a monorepo often enables more deployment flexibility, and more modularity in application structure
Will it be slow?
Rebuilding and retesting everything on every commit is slow. It does not scale beyond a handful of projects. But as I mentioned above, when using Nx, we only rebuild and retest what is affected. Nx also provides build parallelization and distributed computation caching which speeds up a lot of operations in local development and in CI.
Keeping the system malleable is very important when the system’s architecture is still in flux. To do that we need to:
Avoid treating APIs as published unless we have to.
Publish as little as we can as late as we can.
If we don’t follow these, we create published APIs too early. Changing them is very costly, which results in either low dev velocity or in an architecture that doesn’t meet the needs of the system being developed.
Using a monorepo-style development, and a tool like Nx, can help mitigate these problems.
To learn more about public vs published, read the seminal article by Martin Fowler on the subject.
To learn more about Big Ball of Mud, watch this talk by Brian Foot.
To learn more about Nx, check out nx.dev.
To learn more about Nrwl and how we help enterprise software teams build software better, visit nrwl.io.
About the authors
This is a guest post by Victor Savkin – ex-Googler, Nrwl co-founder. Likes monorepos, Nx and Nx Cloud architect, Calligraphy enthusiast, Stoic; tweets @victorsavkin
We build tools for monorepos and help companies use them.