Introduction

As software engineers, we all know that APIs, or application programming interfaces, are essential building blocks for modern applications. From simple CRUD operations to complex integrations, APIs allow us to connect our applications with external systems and services seamlessly. However, designing a robust and scalable API is not always easy, even for experienced developers.

In our latest project, we faced this challenge head-on. We needed to build an API for a new product that even in early prototyping stages could handle a large volume of requests, could be easily maintained, support multiple versions, and be easy to use for developers of all skill levels and its end users. Despite having built dozens of APIs throughout our careers, we still encountered some unexpected roadblocks and made some mistakes along the way. But, as they say, mistakes are the best teachers, and we learned a lot from this project.

In this post, we want to share some of the insights we gained from designing and building our API, and we hope you can find it helpful.

The First Approach: Bottom Up.

When designing an API, it can be helpful to think of it as comprising three distinct layers: the data model, the object model, and the API specification. Without going in too much detail the object model is the representation of the business entities in our problem, the data model is how those entities are designed in a storage mechanism (ex. a database), and the API specification is how developers interact with our object model. In our initial iteration, we began by focusing on the object model, which is often the most enjoyable aspect of API design as it involves creating an abstract model of the business problem (and we developers love to create abstractions of the world!).

Once we had established the object model, we proceeded to implement it using ASP.NET Core and Entity Framework. This allowed us to leverage Entity Framework’s ability to generate the data model (we explored with both CosmosDB and Microsoft SQL Server as our RDBS), including database migrations. We also used Swashbukle (later migrating to NSwag with similar results) to generate the API contract, or OpenAPI specification, this process is also known as Code-First approach.

As explained in What is Code-First? Code-First is mainly useful in Domain-Driven Design. In the Code-First approach, you focus on the domain of your application and start creating classes and controllers for your domain entity rather than design your database or API first and then create the classes which match your database design, or the implementation that matches your API specification. However, during our development journey, we encountered several setbacks that challenged our progress.

Disadvantages of the Bottom Up approach

Collaboration can be difficult

When it comes to rapid prototyping, quick implementation is often a top priority. However, it’s important to keep in mind the potential challenges that can arise in collaborating with other teams, particularly when integrating APIs with other systems.

One challenge we encountered was that there was another team already working to integrate this API with other systems. This created a rocky start for collaboration because the implementation was constantly changing and evolving, making it difficult to establish consistency in the API specification generated by the code.

Even small changes in code adjustments can create frustration during troubleshooting, especially if they happen repeatedly. It’s easy to underestimate the impact of small changes (for example, changing a parameter from nullable to non-nullable) at a large scale, but they can have a significant impact on the efficiency of collaboration.

Consistency is key to building trust, and trust is easily broken in the absence of consistency. Open and constant communication is essential, but it’s not enough when the API is constantly changing. The frequent changes made it difficult to distinguish between design changes and bugs, which eroded trust and made it difficult to establish a stable contract.

In well-established projects, one can address this issue by creating product and API versions that have contract specifications which remain unalterable until the next version is released. However, implementing this approach can be challenging in products that are still in the early stages of design and/or prototyping. Freezing versions is not feasible in such cases since development teams require the latest changes, and there are numerous versions being created each day. Even in very well-established projects, newest versions could require certain level of collaboration where the Code-First approach is not feasible.

To avoid these issues, it’s important to prioritize consistency and establish a stable API specification that everyone can rely on. This means avoiding unnecessary changes and carefully considering the impact of any adjustments on the overall system. By building trust through consistency, teams can work together more effectively and achieve their goals more efficiently.

Framework driving design decisions

The second challenge was also driven by the code driving the data store details. As we start delving into the nitty-gritty details of the Entity Framework implementation we found out that the implementation complexity started to increase. The project presented a non-trivial challenge in the form of implementing hierarchical (tree) relations. However, as with any software project, there will always be non-trivial aspects to contend with. What is significant to note is that our design and implementation decisions were strongly influenced by the Entity Framework, which in turn led to updating the API Contracts to align with the needs of the framework.

So, after two strikes we decided to move to another approach: API design first.

A second approach: API Design first

API design first was not an unfamiliar concept for us. In fact, it’s a crucial part of our Code-With Engineering Playbook, which offers REST API design guidance (if you haven’t already, please take a look at the playbook here). However, there were certain aspects that prompted us to generate the contract code instead of starting with API design.

To illustrate the concept of API design first (and why we decided to go with the Code-first approach), I like to draw a comparison with the OpenAPI specification and how it relates to Infrastructure as Code (IaC). IaC is a powerful tool that enables organizations to deploy and maintain infrastructure consistently and predictably. However, in complex scenarios, it can be overwhelming and present a steep learning curve. Personally, I found it challenging to adopt ARM templates instead of creating resources directly in the Azure Portal and then generating IaC templates from the portal UI. But then, one day, I discovered Bicep, a language for declaratively deploying Azure resources. Bicep is more enjoyable as a developer, easier to maintain than ARM templates, and arguably easier to create resources than directly in the Azure Portal.

In the same way, starting to work directly with the OpenAPI specification was presented as overwhelming and tedious. Then, someone recommended TypeSpec, which was presented as a declaratively approach to document API contracts (which then can be converted to an OpenAPI spec), used extensively by Azure Product Groups. With TypeSpec you can generate very clean, modular and maintainable API specs, where the tool and spec does not get a roadblock, and you can move the conversation to the design itself.

If you are like some of us in the team, and many other software engineers out there, design and documentation is the not so fun part of the software development lifecycle. We understand its importance, but it always felt like a chore. That is what building an API felt like, we just wanted to start coding! That is, before we decided to use TypeSpec. TypeSpec gives you best of both worlds: it allows you to write an OpenAPI specification using code.

Now you may ask, but did we not generate OpenAPI specification from code in the first approach? Is it not the same? The answer is no. Both methods generate OpenApi specification, but in the first approach, the driving force was implementation of the actual application. That then generates the issues we have mentioned around Collaboration and Framework driving decisions. While using TypeSpec, we prioritized on generating the contract, the OpenAPI spec.

Collaboration through Design

Collaboration with other stakeholders through the API contract gets easier, but what about internally? We found out that the collaboration when creating the API contract got also easier by introducing TypeSpec. This was our setup:

• GitHub repo containing the Typespec contract
• GitHub Codespaces with VSCode and default extensions: “bierner.markdown-mermaid” and “Arjun.swagger-viewer”.

Our design sessions usually started by someone driving and sharing their screen, showing in real time the TypeSpec contract generated in the swagger-viewer (or a TypeSpec playground), and doing changes live. Then, every change was reviewed to see how it impacted the data-model and/or the object-model.

These changes were also documented and versioned using mermaid, and we kept relevant design decisions in our Architectural Decision Log. This way we could get back to previous decisions, and also we knew object-model and data-model were consistent. Only then would we start implementing these changes in code.

This approach does not mean that the API design is not informed by the object-model or the data-model (for example, if certain split on tables would be easier to implement than others), but it means that changes were done first in the API design, and then trickle down until the data-model. There were still inconsistences in the road, but it was very different in terms of consistency and trust than with the first approach.

API Contract Versioning

One important aspect of the API contract is versioning. Changes are inevitable, and a good sign of a healthy project is when the contract will keep evolving, the question is how and when to declare a version freeze so the other teams can have a stable base to work. TypeSpec provides ways to do contract versioning through a couple of patterns. If you have used Azure (which by the way uses TypeSpec extensively), you may remember that the APIs have a “YYYY-MM-DD” version format in the query string, we decided to go with the semantic approach (ex. [major]-[minor]-[patch]).

Whenever you publish a new contract make sure to not override previous versions. If you want to see an example of a GitHub workflow that generate versions see TypeSpec Workflow Samples. ASP.NET core will be able to serve several static OpenAPI spec definitions in the Swagger UI.

Adhering to the Contract

Once we had the team rallying around API design first and we had the contract in place, the following challenge was, how to check the actual API implementation followed the contract? How to verify we are actually not breaking the contract?

TypeSpec is not opinionated about how to do it, but the tooling ecosystem around OpenAPI spec was very helpful. After some spike and prototyping we decided to use NSwag command line utility to generate the OpenAPI spec in our CI/CD and then use OASDiff to check for differences and breaking changes in the contract. Eventually we decided to remove Entity Framework, and NSwag stopped working, at that point we had system testing running over a live API, so we decided to modify our CI/CD to generate the contract from the code of that live API, download it and still use OASDiff to check the changes. You can see one example of it in TypeSpec Workflow Samples. We did this, because we wanted to keep using C#, but if language drift is not part of your decision model, you may want to check DREDD.

However, there is another approach worth to take a look. You could generate an SDK Client using Kiota from the API spec and then run all the commands on the generated CLI. If there is an error, it probably means that the implementation it is not according the contract.

Whatever technology or tooling you use to check the contract, it is important to use one, and integrate it to the CI/CD. It is better to detect early a drift, than later when you have already built tooling around the spec.

While we tried to share our journey it is important to synthetize what API design first is, and what it is not:

What API Design First is?

API design first is a methodology that prioritizes designing the API contract before writing any code. It involves defining the data structures and API endpoints that will be used by the application, with the goal of creating a clear and consistent interface that meets the needs of both the client and server. Some key aspects of API design first include:

• Defining the API contract first, before writing any code.
• Focusing on creating a clear and consistent interface that meets the needs of both the client and server.
• Prioritizing collaboration with stakeholders, including developers, product owners, and designers.
• Using tools like TypeSpec to help generate and maintain the API contract.
• Versioning the API contract to provide stability for other teams.

What API Design First is NOT?

While API design first can be a powerful methodology for creating effective APIs, it’s important to be aware of what it is not. Some common misconceptions include:

• API design first is not just about creating documentation: While documenting the API is an important part of the process, API design first is about creating a contract that serves as the foundation for the API implementation.
• API design first is not just for large projects: Even small projects can benefit from API design first, as it helps ensure that the API is consistent and easy to use.
• API design first is not just for RESTful APIs: While many of the tools used in API design first are geared towards RESTful APIs, the methodology can be used for other types of APIs as well.
• API design first is not a one-time process: The API contract will evolve over time as the needs of the application change, and it’s important to continue prioritizing collaboration and versioning to ensure consistency.
• API design first is not a specific tool, although we are very opinionated with using TypeSpec, there are many ways to document the API, either by tooling or methodology (ex. mermaid, UML use cases, cucumber, OpenAPI spec, etc.).

Conclusion

API contract-first design is an approach that can significantly improve the consistency, reliability, and maintainability of your APIs. By using a tool like TypeSpec to create and manage your API contracts, you can involve stakeholders from different teams, reduce errors and inconsistencies in your data and object models, and improve the versioning and documentation of your APIs. Additionally, integrating a tool to check the API implementation against the contract can further improve the quality of your APIs and help you catch errors early in the development process. Overall, adopting a contract-first approach can help you build better APIs that are easier to use and maintain, and that can provide value to your customers and stakeholders.

Acknowledgments

The team behind the journey: Emmeline Hoops, Yani Ariunbold, Olha Konstantinova, Marcia Dos Santos and Wendy Reinsel. Special thanks to the Azure TypeSpec team for the support, and to our main stakeholders Dan Massey and Bart Robertson for their active involvement and support in our learning journey.