Behind the scenes of the Nightscout API

Mark W.

Darrel Miller

Like peanut butter and chocolate, it’s awesome when two great things come together. This post originated as part of the Hack Together: Microsoft Graph and .NET. You can get more details and participate by at the registration link. Enjoy!

Developers today are building ever more complex apps that, increasingly, apply capabilities from a wide range of services. One common practice is weaving together the capabilities of multiple different apps to create something new and unique. In this scenario, let’s call it an example of IoT predictive maintenance, we wanted to build an application that could remind someone to check their glucose monitor by putting a reminder on their outlook calendar. While this scenario is rather simple, it requires integration with Microsoft Graph, the Glucose monitor, and user authentication for both systems, which use different security schemes. You can see how even in this simple example, things get complex rather quickly!

For developers, APIs are the engine that drives applications that pull data from multiple services. OpenAPI is the industry standard specification that is a developer’s best friend when it comes to integrating platforms. To build this app, we need to use APIs for both Microsoft Graph and the Glucose monitor. But despite being a standard, there are still many ways to express the same thing. Consider that for each operation the developer needs to specify the HTTP method, headers, the path, query parameters, return codes, valid values, optional and required parameters, etc. This detail must be provided both the request and response. Then, most importantly, the developer must describe the information–the types–that their service provides.

Given the flexibility of OpenAPI, it’s easy to see how different developers–even within the same organization–can create specifications that are similar, but different. Because the OpenAPI specification is the “contract” for the service, how the API is declared can significantly affect downstream toolchains. For example, some of the initial design decisions of the Nightscout API made it difficult to apply the code generation capabilities of Kiota](https://microsoft.github.io/kiota/get-started/). We chose to “refactor” the original specification using TypeSpec (née Cadl) to feed a more precise API definition into our client code generator, Kiota. The complete set of code for this example is located in the Nightscout Description repository in the APIPatterns organization in GitHub.

To become more familiar with TypeSpec, please check out the TypeSpec Docs and the TypeSpec playground. The following blogs, The Value of TypeSpec in designing APIs, Describing a real API using TypeSpec: The Moostodon Story shows another example of using TypeSpec to describe APIs and Kiota to generate client libraries.

This scenario requires access data from Microsoft Graph and a glucose monitor. For our team, how to get data from Microsoft Graph is well known and something we do everyday. Glucose monitoring is new to us, so we went looking for an OpenAPI description… and found one! We immediately ran the OpenAPI description through Kiota to create our dotNet client, and were “bitten” by some design decisions made by the original developer of the Nightscout API.

The Nightscout API has a discriminator as a required parameter on the path. The issue is that OpenAPI doesn’t support using a path parameter as a discriminator. As a consequence, Kiota isn’t able to properly generate client libraries. The OpenAPI was likely described using a discriminator because the API supports many different document types, each with the same API capabilities. Writing OpenAPI to fully specify endpoints, each with the same capabilities, requires duplicating all of the operations for each document type. Authoring an API without discriminators is error prone, time consuming, and results in a large document. Enter Cadl, err… TypeSpec!

TypeSpec is an open-source language inspired by TypeScript that’s designed to make the authoring of APIs easier and less cumbersome. Originally called Cadl (pronounced “cattle”), the team is in the process of renaming the project to TypeSpec to give it a more accurate and descriptive name. Because TypeSpec is a language, TypeSpec has better capabilities to reuse API designs and separating concerns, making the generation of complex OpenAPI documents incredibly easy. However, we still like cow puns, so, let’s round up the herd and do some refactoring!

Create the models

We used the original Nightscout API as the basis for our refactoring, and expressed its APIs using TypeSpec. We start by defining the models (also known as types) that are used by the service. In our example, the models are located in the ./spec/models folder. We capture common properties in a “base” model, and then extend it for specific document types. The common properties for Nightscout documents are factored out into the DocumentBase.cadl file, which is imported–just like code–when we model each individual document type. For example, here’s the first part of the Food.cadl file:


import "./DocumentBase.cadl";

@doc("Nutritional values of food")
model Food extends DocumentBase {
@doc("food, quickpick")
food?: string;

@doc("Name for a group of related records")
category?: string;

Define a reusable interface

The next step was to address the main issue of getting rid of the discriminator in the path. In the original OpenAPI specification, the result is defined as oneOf a specific type, for example, Food. The {collection} discriminator in the path determines which set of documents to query and, as a result, the type that is returned. Essentially, we need to remove the ambiguity in the path and change /{collection}/{identifier} to /food/{identifier}. If we look closely, we realize the reason a discriminator in the path could be used is because the operations on each collection are identical. In TypeSpec, we can group operations into an interface, then reuse it across multiple endpoints. Being able to specify the exact shape of multiple endpoints in a single definition, and then being able to apply that definition to multiple endpoints, is a powerful technique for driving standardization and consistency across a broad API surface area. In the Nightscout example, the operations on collections, are captured in the ./spec/documentCollection.cadl file. Here, a single DocumentCollection interface is defined that contains all the CRUD operations on collections. The following example shows how PATCH is expressed. The <T> represents a template, and is replaced with a specific model type when the interface is used.


interface DocumentCollection<T> {
...
   @summary("PATCH: Partially updates document in the collection")

   @patch 
   op patch(
           @header("If-Modified-Since") ifModifiedSince: string;
           @path identifier: string,
       ): createdDocument | updatedDocument | BadRequestFailedResponse | UnauthenticatedFailedResponse | UnauthorizedFailedResponse | NotFoundFailedResponse | GoneFailedResponse | StatusResponse<412> | StatusResponse<422>; 

All of the endpoints return a JSON object that contains a status property that duplicates the HTTP status code. The actual response body is in the results property. How responses are modeled is captured in the ./spec/responses.cadl file. In similar fashion, a common model is defined StatusResponse<Status>, and then “instances” of those models are created, which can have additional properties. In the previous example, the PATCH operation (op patch), returns one of these declared responses, for example, createdDocument OR (|), BadRequestFailedResponse, OR StatusResponse<422>.

Not only can we pass in a specific HTTP return code if necessary, but we’re able to use different response types to accurately model service behavior. When attempting to create a document, if it exists, the service returns a different response body. A developer must carefully read the OpenAPI document to understand they must evaluate the response code, 200 or 201 to determine if a document is created or updated. In TypeSpec, it’s easier to indicate creation versus update, and is modeled as follows:

model createdDocument is StatusResponse<201> {
   @header("Location") location: string;
   identifier: string; 
   lastModified: int64;
}

model updatedDocument is StatusResponse<200> {
   identifier: string; 
   isDeduplication?: boolean;
   deduplicatedIdentifier?: string;
}

It’s fairly uncommon for APIs to describe the 200 and 201 response as two distinct response bodies, however, it’s a perfectly valid API design. Kiota doesn’t have a great solution for this particular scenario at the moment, but with the use of the AdditionalData property, all of the returned information can be accessed.

Now, we have our models, a common interface, and a standard set of responses. All we need to do is declare the endpoints in our API. We use the @route decorator to establish the path segment. We get the operations by declaring our route is “decorating” new interface that extends our common DocumentCollection. Instead of a discriminator, the kind of document collection accessed is explicitly expressed through the template parameter, <T> . For example, here’s the endpoint for Food:

       @route("food")
       interface foodDocument extends DocumentCollection<Food> {}

Work with multiple versions of an API

We also had some other interesting discoveries, one of which was that not all of the capability that we need is in the V3 API. We were easily able to include select operations from V2, and keeping them isolated in their own namespace. In TypeSpec, namespaces work much like they do in code, and provide the same organization and isolation mechanism for APIs. When we generate client code using Kiota, the result is a single library that includes operations from both versions of the API. Having a single library that works with both versions API makes is easy for developers to write code that uses the service.

@service({
   title: "Nightscout API",
   version: "3.0.3"
})
@useAuth(AccessToken | JwtToken)
@server("/api","default endpoint")
namespace nightscout {
   @route("v2")
   namespace v2 {
       @route("properties")
       op getProperties(): Properties;

       @route("properties/{properties}")
       op getSelectedProperties(@path properties: string[]): Properties;
   }
   @route("v3")
   namespace v3 {
       @route("devicestatus")

Separate concerns using “sidecars”

The original OpenAPI description contains lots of usage documentation about the API and its operations–which is fantastic. However, it’s common for many people to work on an API. For example, a developer creates the operation definitions, while a product manager might write the documentation. TypeSpec, through a concept called sidecars, facilitates a clean separation of concerns. In our example, we factored out the documentation into a distinct file, ./spec/docs.cadl. Using the @@ construct, we were able to index into another Cadl file. In our example, we added documentation to the read operations of our common interface:

@@doc(DocumentCollection.read, """
Basically this operation looks for a document matching the `identifier` field returning 200 or 404 HTTP status code...

Generate OpenAPI and client libraries

When complete, our main.cadl file is a concise 66 lines of code, making it easy for a developer to quickly understand the entirety of an API. Taking into account the models, common interfaces, and the documentation, the entire TypeSpec totals around 500 lines of code. When rendered as OpenAPI, the resulting specification is over 5,000 lines of code! By generating the OpenAPI from TypeSpec, we get a specification that conforms to our practices and guidelines. Using TypeSpec to codify guidelines, practices, and patterns to generate cleaner, more consistent specifications, is exactly what the Microsoft Graph and Azure SDK teams are doing!

Now that we have a newly constructed OpenAPI, creating a client library is a single command away with Kiota. You can install the Kiota command line tool using the instructions at https://aka.ms/get/kiota. Using Kiota developers can generate client libraries in C#, Go, Java, TypeScript, Python, and Ruby.

kiota generate -l <language> -o <output path> -d https://raw.githubusercontent.com/apidescriptions/nightscout/main/spec/cadl-output/%40cadl-lang/openapi3/openapi.yaml

With this client library, you get a strongly typed experience for accessing the API with all the capabilities we built to make Microsoft 365 applications resilient and efficient.

A final shout out

As you can see from our previous TypeSpec blog posts (Moostodon and The value of Cadl in designing APIs), we’ve been having fun with TypeSpec and Kiota. The combination of the two is proving to be a powerful and elegant way to bring the best developer experience to the authoring OpenAPI specifications and quickly generating client code. We know you want to be “herd”, so let us know your thoughts and take a moment to try out TypeSpec. Check out the code in the APIPatterns organization, and become part of the “moovement!” Sorry, we miss ‘Cadl’ because we just can’t resist bad cow puns.

And finally, there were many other folks that contributed to this demo and blog who deserve recognition: Vincent Biret, Mike Kistler, Sébastien Levert, and Rabeb Othmani. And of course, Scott Hanselman, who came up with some of the original ideas and prototype. Thanks for all your help!

3 comments

Discussion is closed. Login to edit/delete existing comments.

  • Michael Taylor 0

    While I find this article interesting, and the CADL/Kiota ones before it, I really wonder if there is any value add to this entire stack for anyone outside Microsoft, Amazon and a few other providers. OpenAPI is a spec for describing an API, its inputs and outputs. It is basically an IDL that can be used to auto-generate clients and potentially stub out servers. Typespec/CADL seems to be a spec for defining a structure to define an IDL that can then be used to auto-generate clients/servers. Why? At what point do we need a spec to define the overly complex Typespec to define the structure of the IDL that can then be used to auto-generate client/servers?

    The gist of Typespec to me is to provide a more type-safe (and slightly OOD) like description of an API. This is sort of what OpenAPI was trying to do, minus the type safety. Wouldn’t there be more value add to simply expand the OpenAPI spec to be more type-safe/OOD? The less stuff that needs to be generated to get from a definition to the client/server the better. I tried to describe our simple 6 endpoint API in CADL and it was a real painful experience. The syntax is non-intuitive and you still end up having to then look at the OpenAPI generated to confirm it lines up with what you want the final API to look like. So instead of having to understand OpenAPI and its rules you now also need to understand Typespec’s rules. I would much prefer to simply have OpenAPI++ which expands OpenAPI with features to make it easier to get things that Typespec is offering such as reusable components. OpenAPI does support reuse but inheritance and some other features it is missing.

    I also tried to use Kiota and honestly it is probably the worst API client I have ever seen. It makes AutoRest, which I strongly dislike, look like a great option. The way Kiota exposes an OpenAPI pretty much stuffs it in your face. I cannot remotely imagine writing code that looks like `client.V1.Users[“10”].PutAsync` that updates a user account with ID 10. This may be semantically correct for what the API is doing but it is not readable so every API call would need some sort of commenting about what it is actually doing. Or more likely you’ll want to wrap API clients in a “service” class. But now we’re creating wrappers around a client that mostly do nothing but put friendly names on the front. This is a step back in my opinion. Kiota might be useful as a very-low level layer on top of an HTTP client but there is still work to convert that low level client into something an app would want to use. So I see it as just an unneeded layer. I really hope Azure SDK doesn’t switch to this generator. I even looked into seeing whether Kiota could be extended to do what I want and it is pretty much impossible at this point because the generators are not extensible. Even making it so it use the operation ID from OpenAPI isn’t doable at this point.

    I also considered creating my own generator based upon the existing C# generator but the Kiota tool hard codes supported generators so you have to write your own generator and then update the main Kiota tool to know about it. At this point Kiota seems pre-mature as a client generator. I would really like to see the blog focus on a more powerful client generator.

    • Darrel MillerMicrosoft employee 0

      Hey Michael, I appreciate your feedback and you raise a lot of good points. In theory, OpenAPI could be extended to incorporate some of the capabilities of TypeSpec. However, one of the strengths of OpenAPI is the broad community of tooling that supports it. Every new capability that gets added to OpenAPI puts a burden on those tooling creators to add support for it. We in the OpenAPI technical developer community also get a significant amount of feedback that OpenAPI has increased in complexity over the years and there is reluctance to add more. I can assure you that we have spent many, many hours tried to reach consensus on additional reuse capabilities in OpenAPI that are toolable and don’t compromise the authoring experience. It is a hard problem to solve. TypeSpec has the advantage of being able use a language style syntax to create cleaner and more expressive constructs over what is possible in a YAML/JSON format. However, TypeSpec syntax and extensibility via decorators means that it is harder for other tooling to consume. Personally, I think the combination of TypeSpec and OpenAPI provides the best of both worlds. TypeSpec gives the great design experience that has great tooling support and OpenAPI has interoperability benefits.

      I can understand your reaction to Kiota. Many people have the expectation that a code generator should be customizable to produce a API surface that they find appealing. However, Kiota’s philosophy has been to take the approach of being rigorously consistent in its mapping of HTTP requests to the API surface area. Our goal is to produce an API surface area that will work for any HTTP API surface area of any size, that can evolve in a non-breaking way as the API surface area grows. The API surface maps directly to the HTTP path structure and so does not use the OperationId as a method name. OperationIds can work well for smaller APIs but there are challenges for large API surface areas. We believe that developers design API path structures with intent, and it is natural to carry that intent through to the client experience in the API consumers native language. It is a perfectly reasonable reaction to say that is not a style that you want to use throughout your application and one of the reasons we recommend creating a service wrapper to expose just the API functionality your application needs as a point of decoupling. I listed a number of other benefits of the service class in the example application.

      API style is a somewhat subjective matter and it is challenging to build client code generators that attempt to generate client code that suits every taste. As an alternative, we generate consistent code that is predictable that you can hide behind a facade that suits you. We will take care of constructing URLs with the proper encodings, serializing types quickly and reliably, and all the other cross cutting concerns like observability, rate limiting and authentication.

      I understand that you see the lack of extensibility as shortcoming. However, we in the Microsoft Graph team consider consistency as a critical part of the developer experience for APIs and so have chosen not to allow Kiota to vary its outputs. That design choice may not suit you, and that’s ok. Being able to pick and choose the tools that we want to use is the best part of open ecosystems. And Kiota is OSS, if you don’t like what it produces, fork it and build something that does suit you.

      Cheers,
      Darrel

    • Mark W.Microsoft employee 0

      Michael, thanks for the checking out the blog and the feedback. Darrel’s pointed out a number of good points about Kiota, so I’d like to expand a bit more on why TypeSpec is proving valuable to our team. Part of this is captured in the post that Mike and I wrote about the value of creating a TypeSpec (nee Cadl) library. We established an API Stewardship Board and regularly work with teams design their APIs to help them understand good design and apply our REST API Guidelines. At Microsoft, we have hundreds of teams building cloud services. We strive to provide APIs that are consistent, intuitive, easy to use, and version resilient. Given the sheer amount of services, and number of developers, this is a significant challenge. And you might be inclined to think these problems that exist only at companies the size of Microsoft. However, after talking with dozens of developers at companies a fraction of the size of Microsoft, I can assure you, we are all struggling with this challenge!

      We’ve seen that developers don’t fully understand the nuances how their service “maps” to an HTTP based API. There are a lot of things for a developer to consider, for example, how to properly express an idempotent operation or model a long running operation. Now layer on top of the nuances of HTTP, our REST guidelines, e.g. how errors must be modeled, using PATCH (with a content type of application/merge-patch+json) for updates, etc. And finally, let’s try to avoid breaking changes. For example, because client code must write iterators on collection, adding a nextlink is a breaking change. So we ask our service developers, “yes, your collection will not return a lot of results now, but what about in five years? Is your collection ever going to return enough results to be pageable?” Many of these things can’t be caught with a linter. And this is where TypeSpec is becoming a valuable part of our toolchain.

      Using TypeSpec, we can more accurately codify our guidelines and idioms into a reusable library that developers can start with to model their API. It allows them to think more about their core resources (models) and how they will be used, and less about the nuts and bolts of how they should be represented in OpenAPI. For example, when a developer needs a long running operation, all they need to do is add to the resource the @pollingOperation decorator. TypeSpec handles creating the ~80 lines of OpenAPI that will be generated exactly how our REST Guidelines specify. (Here’s a link to an example.)

      Please keep in mind that TypeSpec is still new. And you’re right–this is something that developers will have to learn. It’s similar to TypeScript, so if this is something you are not familiar with, the syntax might seem a bit odd. However, the alternative is that developers have to learn all the nuances of OpenAPI, their organizations REST API Guidelines, breaking change policy, and style guide. We’ve seen that devs don’t always do this and, instead of taking a design first approach, they fall back to code first approaches, like swashbuckle, to generate their OpenAPI document. From first hand experience, I can tell you this ends up causing a lot more work.

      If you are interested in sharing your thoughts on TypeSpec syntax, I work with the TypeSpec team on a daily basis and they are great folks who are always open to feedback. Both Darrel & I participate in the OpenAPI Technical Developer Community, which is open to anyone and meets weekly to work on a range of issues, including the spec. In fact, we are currently working on some ideas for the next version of OpenAPI, so this could be an opportunity to raise your ideas about ‘OpenAPI++’.

      All that said, whether it’s TypeSpec, Kiota, AutoRest, or something else, you need to find the right tools and processes that work for you and your team. It would be great to learn more about your toolchain, the challenges you face building APIs, and ideas on improving OpenAPI. Maybe a post on Medium??

      Thanks again for taking the time to give us feedback,

      -Mark W.

Feedback usabilla icon