Describing a real API with Cadl: The Moostodon story
Describing a real API with Cadl: The Moostodon story
In our previous blog post, The value of Cadl in designing APIs, we demonstrated the different capabilities of the Cadl API design language. We walked through the evolution of simple API design to take advantage of the reuse, partitioning, and refactoring that’s possible with Cadl and its tooling.
Our previous post used an artificial “WidgetService” API to show off the capabilities of Cadl. In this post, we’ll demonstrate many of the same features but using a real API. We’ll show how we created a description of the Mastodon API using Cadl. We’ll also show how we used the Cadl and Kiota tooling to generate client libraries for the API in multiple languages.
What is Mastodon?
Mastodon is a federated social network. It’s a decentralized, open source alternative to commercial platforms like Twitter and Facebook. The Mastodon API is a REST API that allows applications to interact with Mastodon instances. It’s a rich API that allows applications to do things like post toots, follow other users, and search for users. It also allows applications to interact with the underlying data model of Mastodon. For example, it allows applications to create new accounts, configure instances and manage the other instances a particular service interacts with.
Mastodon uses OAuth2 for authentication, requiring that applications must first register with the Mastodon instance they want to interact with. The Mastodon instance will then provide the application with a client ID and client secret. The application can then use these credentials to request an access token from the Mastodon instance. The access token can then be used to authenticate requests to the Mastodon API.
Why Moostodon?
One of the curious aspects of the Cadl name is that many people hear “cattle” when they first hear it pronounced. To maintain the bovine theme and a certain amount of Canadian influence, Moostodon seemed like a suitably silly name for this sample. We’re working on a new “official” name for the Cadl project, but it’s our hope that bad cow puns will never be put out to pasture.
State of the union
As far as HTTP APIs go, the Mastodon API is sizeable. It has more than 100 endpoints and many different types of resource models. It uses HTML forms for sending data and JSON payloads for returning data. A contributor to the Mastodon project recently submitted an OpenAPI description, but at more than 5,000 lines, there’s reluctance by the project to accept as it will require maintaining. Another issue is proposing the use of tooling to generate the OpenAPI description from the code as a more maintainable approach. While this strategy works for some projects, it isn’t a design-first approach that’s practiced by many teams. Cadl is a design-first language that’s more maintainable and generates OpenAPI to take advantage of the ecosystem of available tooling.
Divide and conquer
Due to the size of the API, the first step in creating the API description was to divide the API into smaller pieces to make functionality easier to find.
One of the strengths of Cadl is that it allows developers to structure their API in a more code-like manner. The API design can be divided into separate files based on function—a classic design technique in traditional programming languages. A top-level namespace imports the various components into the full service definition. The @service decorator on the top-level namespace provides service metadata.
To learn more about the Cadl language and decorators, see Getting Started with Cadl.
@service({
title: "Mastodon",
version: "1.0.0"
})
@route("/")
namespace MastodonApi {
}We attach the @route("/") decorator to the namespace so that all operations directly inside the MastodonApi namespace will be hosted at the root of the API.
In the following example, the OAuthService is hosted at the /oauth path. The definition of the OAuthService is in a separate file that’s imported into the main API description.
@route("/")
namespace MastodonApi {
@route("oauth")
interface oauthResource extends OAuthService {}
@route("api/v1")
namespace v1 {
@route("accounts")
interface AccountsResource extends AccountsService {}
}
@route("api/v2")
namespace v2 {
@route("search")
interface SearchResource extends SearchService {}
}
}Over the years, the Mastodon API has evolved and some resources have been replaced by a V2 API. However, only some of the resources exist in the V2 API so it’s necessary to describe both the V1 and V2 APIs. Cadl allows us to group these using nested namespaces. Cadl does have a rich versioning strategy, which will be covered in another post, but in the interest of simplicity, we’ll add v2 as a new namespace.
Reusable API patterns
Many of the Mastodon resources follow a regular pattern of operations that provides the opportunity for reuse. For example, the MutesResource, BlocksResource, and EndorsementsResource all provide the same operations on a set of Account resources. Just like code, we can refactor these operations into an interface (NamedSet) that can be reused in all three places. Because Cadl has a VS Code plugin, we’re guided through this process with IntelliSense (and GitHub Copilot). Since errors are caught by the Cadl compiler, we have a high degree of confidence that our refactoring is correct. This style of reuse can be seen in the following example:
@route("api/v1")
namespace v1 {
@route("mutes")
interface MutesResource extends NamedSet<Account> {}
@route("blocks")
interface BlocksResource extends NamedSet<Account> {}
@route("endorsements")
interface EndorsementsResource extends NamedSet<Account> {}
}Describe operations
The definition of the AccountsService interface is in a separate file that is imported into the main API description using import "./services/accounts.cadl"; at the top of the main Cadl file.
interface AccountsService {
// Register a new account.
@post createAccount(
@body body: CreateAccountForm
): Account | UnauthorizedResponse;
// Search for accounts.
@route("search")
@get search(
@query q: string,
@query limit: int32,
@query following: boolean,
@query resolve: boolean
): Account[] | UnauthorizedResponse;
// Find familiar followers
@route("familiar_followers")
@get getFamiliarFollowers(
@query id: string[]
): FamiliarFollowers[] | UnauthorizedResponse | UnprocessableContentError;
}This interface is limited to the operations that are either directly available at the /api/v1/accounts path or are available at a subpath of that path. Cadl doesn’t constrain how you group your operations. The approach followed here tries to limit each interface to a reasonable number of operations. As organizations gain experience in designing APIs with Cadl, they’ll likely develop their own best practices.
Reuse types
While there are sometimes opportunities to reuse interface patterns in an API, there are almost always opportunities to reuse type definitions in an API. Cadl has various ways to reuse types.
New models can be created based on other models, and then have other decorators applied. In the following examples, some common errors are defined so they can be reused with friendly names.
model UnprocessableContentError is Error {
@statusCode statusCode: 422;
}
model UnauthorizedError is Error {
@statusCode statusCode: 401;
}Cadl has the interesting ability to combine the use of a model and the spread operator to define reusable sets of parameters. The following RangeParameters are used in many places in the Mastodon API.
model RangeParameters {
@query max_Id?: string;
@query sinceId?: string;
@query min_Id?: string;
@query limit?: int32;
}
interface NamedSet<T> {
@get items(...RangeParameters): T[];
}
interface TimelinesService {
// Get public timeline
@route("public")
@get publicTimeline(
@query local : boolean,
@query only_media : boolean,
...RangeParameters
): Status[];
// Get Home Timeline
@route("home")
@get getHomeTimeline(
...RangeParameters
) : Status[] | UnauthorizedResponse | NotFoundResponse;
}Mastodon is unusual for HTTP APIs in that it uses application/x-www-form-urlencoded for most of its update operations. To indicate that a model will be sent as a form, the @header decorator is used to indicate the content type. The Form model was created as a template for all of the form models that are used in the API. In this case, the is operator was used instead of extends to prevent an allOf being generated in the OpenAPI document because Kiota interprets allOf to represent inheritance.
model Form {
@header contentType: "application/x-www-form-urlencoded";
}
model TokenForm is Form {
grant_type: string;
code: string;
client_id: string;
client_secret: string;
redirect_uri: string;
scopes: string;
}All the examples shown here are excerpts from the complete Cadl description in the Moostodon GitHub repo. Using this description, it’s simple to create an OpenAPI description by compiling the Cadl from the spec folder.
cadl compile main.cadlCadl and OpenAPI
The Moostodon project has the Cadl files in the spec folder, along with the generated OpenAPI document. Cadl creates artifacts through the concept of emitters. The OpenAPI emitter will create a specification compliant with OpenAPI V3.
Looking at the main.cadl file, we can immediately see many benefits of using a design-first approach with Cadl:
- The concise Cadl description is much easier for developers to understand, compared with the OpenAPI document.
- Core concepts are encapsulated and reused. Just like developers understand how to use class libraries, they can now understand how to use API patterns.
- Reuse is inherent in the language through libraries.
- Cadl is flexible and can easily model complex APIs.
Now that we’ve generated an OpenAPI document for our Moostodon service, we can use it to drive downstream tool chains. One common use case is to generate client code. Let’s do that now!
Create a client
There’s an entire ecosystem of tooling built around OpenAPI documents. You can find many of them listed on the OpenAPI Tools page. There are many client SDK generators that can be used, but for this example we used Kiota.
Why Kiota?
Similar to AutoRest, which was developed by the Azure SDK team, Kiota is an open-source project developed by the Microsoft Graph Developer Experience team that is optimized for the API consumer experience. Because Microsoft Graph is a large API, we found that nobody builds an application that needs to call all of the APIs. Kiota enables selecting just the parts of the API the client application needs to call and generate out a client for their needs. This approach also lowers the barrier of learning a new SDK every time you need to call a new API that is described by OpenAPI. Because Cadl generates OpenAPI documents that adhere to our standards, Kiota becomes more powerful. The two tools together promote a design first approach without sacrificing the ability to get to code quickly.
Tour the Kiota clients
There are three example clients in the clients folder of the GitHub repository. The C# one is more developed than the Python and Node.js one, but they can all access public Mastodon APIs that are read-only. The C# example can also get authentication tokens and perform write operations.
The generated client code can be found in the sdk subfolders for each language and can be generated using the following command from the respective language folder:
kiota generate --language typescript --output sdk -d ..\..\spec\cadl-output\openapi.json --clean-output --class-name mastodonClient
kiota generate --language python --output sdk -d ..\..\spec\cadl-output\openapi.json --clean-output --class-name mastodonClient
kiota generate --language csharp --output sdk -d ..\..\spec\cadl-output\openapi.json --clean-output --class-name mastodonClient --namespace-name MastodonClientLib --structured-mime-types application/x-www-form-urlencoded --structured-mime-types application/json --structured-mime-types text/plainThe variety of options provided by the Kiota generate command enable configuring the features that are supported by the client code. It would be inconvenient to memorize all these options each time the OpenAPI description is updated and the client code regenerated. Kiota solves this problem by preserving the configuration options in a kiota-lock.json file, which is stored in the output folder. For that reason, performing updates in the future is as simple as:
kiota update --output sdk --clean-outputThe generated code does have dependencies on some small core libraries that provide the native HTTP capabilities, serialization and authentication support. You can discover what the necessary dependencies are with the kiota info -l <language> command. Other than the single Kiota Abstractions library, all of the other dependencies are replaceable by custom implementations.
Isolate the API client calls
The generated client code is a great starting point, but using it directly isn’t the ideal way of integrating it into a production application. The first thing to do is to separate the client code into a service class. This pattern is common in application architecture and isolates the application logic from the mechanics of calling a specific API. It provides a great place to put an interface to enable testing your application with a mock API service. The service class can also be used to manage the authentication token and to provide a higher level API to the client application. The following code shows the service class for the C# client:
public class MastodonService {
private MastodonClient client;
private OAuth2AuthorizationProvider _authProvider;
public MastodonService(string baseUrl)
{
_authProvider = new OAuth2AuthorizationProvider(
CredsHack.ClientId,
CredsHack.ClientSecret,
"urn:ietf:wg:oauth:2.0:oob",
baseUrl);
// Use native HttpClient as the underlying HTTP library
var requestAdapter = new HttpClientRequestAdapter(_authProvider);
// Add support for form-urlencoded content type
SerializationWriterFactoryRegistry
.DefaultInstance
.ContentTypeAssociatedFactories.Add(
"application/x-www-form-urlencoded",
new FormSerializationWriterFactory());
client = new MastodonClient(requestAdapter);
_authProvider.Client = client; // Enable auth provider to use client
requestAdapter.BaseUrl = baseUrl; // Overwrite baseUrl from OpenAPI
}
...
}We can create methods in the MastodonService class that perform the API functions our application needs with a simplified interface for the use-case. An interesting side effect of this pattern is that once you’ve done a couple of these helper methods, GitHub Copilot does a great job of implementing the methods from a simple descriptive comment.
// Get a specified account's followers
public async Task<List<Account>> GetFollowers(string id, CancellationToken cancellationToken = default)
{
var followers = await client.Api.V1.Accounts[id].Followers.GetAsync(cancellationToken: cancellationToken);
return followers;
}More media types
Currently, Kiota supports serialization of JSON and text content out of the box. We’ll add support for more media types in the future. In the meantime, you can add support for another media types by adding a SerializationWriterFactory to the SerializationWriterFactoryRegistry. The following code shows how the Moostodon application added support for the form-urlencoded content type:
// Add support for form-urlencoded content type
SerializationWriterFactoryRegistry
.DefaultInstance
.ContentTypeAssociatedFactories.Add(
"application/x-www-form-urlencoded",
new FormSerializationWriterFactory());After registering your new serialization factory, don’t forget to tell Kiota that you now support this media type by adding it to the --structured-mime-types option when you generate the client code.
Mastodon uses form-urlencoded content type write operations. By adding support for this media type and describing in the Cadl/OpenAPI the use of that content type, the generated client code knows how to serialize the CreateAppForm object into the correct HTTP content.
// Create an app
public async Task<Application> CreateApp(string appName, string redirectUri, string scopes, string website, CancellationToken cancellationToken = default)
{
var app = await client.Api.V1.Apps.PostAsync(new CreateAppForm() {
Client_name = appName,
Redirect_uris = redirectUri,
Scopes = scopes,
Website = website
}, cancellationToken: cancellationToken);
return app;
}It’s this kind of abstraction that massively simplifies the developer effort. The HTTP APIs can use the best media types for the scenario and the client developer works with types that are most natural to them. The generated client code handles the mapping.
Authentication is unavoidable
Adding a method to post a status is also seemingly simple:
// Post a status
public async Task<Status> PostStatus(string status, CancellationToken cancellationToken = default)
{
var newStatus = await client.Api.V1.Statuses.PostAsync(new CreateStatusForm() {
Status = status
}, cancellationToken: cancellationToken);
return newStatus;
}However, in order to post a status, the user must be logged in. Mastodon supports two different types of authentication flows. One is the OAuth2 Client Credentials flow that’s used to acquire a token for an application. The other is the OAuth2 Authorization Code flow that’s used to acquire a token for a user. Kiota has a plugin model for authentication providers. We’ve implemented the OAuth2AuthProvider to enable the generated API client to request a new token when a request is made. Handling authentication with a plugin keeps the code that’s calling the API separated from token-handling code. This approach is illustrated in the above example that posts a status on behalf of a user but has no authentication-related code.
The following code shows two methods that the MastodonService class exposes to enable the application to get the right type of token before making an API call.
internal async Task LoginApp(CancellationToken cancellationToken = default)
{
await _authProvider.LoginApp(cancellationToken);
}
internal async Task LoginUser(string username, CancellationToken cancellationToken = default)
{
var url = _authProvider.GetUserAuthorizationUrl("write");
//Display the url to the user and ask them to enter the code
Console.WriteLine("Please open this url and sign in and copy code into console: " + url);
Console.Write("Enter code: ");
var code = Console.ReadLine();
await _authProvider.LoginUser(code, cancellationToken);
}Moostodon is a console application, and very much a demo application, so we were able to take certain liberties. To acquire user tokens, we’ll generate an AuthorizationUrl that can be opened in a browser. In the browser window, the can user sign in, consent to the specified scopes, and acquire an authentication code. That authentication code can be used to acquire a token.
The current implementation of the OAuth2AuthProvider only holds a single token and so it’s necessary to perform the appropriate form of sign-in prior to making any authenticated API calls. Building a production quality OAuth2AuthProvider is out of scope of this sample.
Languages, languages, everywhere
As developers, we have no shortage of choices when it comes to selecting a programming language. Thankfully, HTTP is a universal language. We can use the same HTTP APIs from any language. Kiota currently has support for many languages, and you can always see what’s supported with the info command.
[
In the Node.js TypeScript example, the pattern of creating a MastodonService is followed. The service methods are similar to the C# project’s methods. The Node.js example doesn’t yet have an OAuth2AuthProvider and so only publicly accessible examples are included:
// Get the mastodon public timeline
public async getPublicTimeline(): Promise<Status[] | undefined> {
return await this._client.api.v1.timelines.public.get();
}The Python example isn’t different:
def get_public_statuses(self) -> List[Status]:
"""
Gets the statuses from the Moostodon API.
Returns:
The list of statuses.
"""
statuses = asyncio.run(self.client.api().v1().timelines().public().get())
return statusesTools in the toolbox
Cadl and Kiota are new tools being developed by the Azure and Microsoft Graph developer experience teams to help developers produce and consume HTTP APIs. This post has hopefully demonstrated how these tools can be used with any HTTP API, regardless of the language or platform. We’re excited to see what you build with these tools.
Check out the code and toot us at @weitzelm@hachyderm.io, @darrel_miller@masotdon.social, and @mikekistler@mastodon.social.
Let’s start a moovement!
Mark W. Principal Architect, Developer Division
Mike Kistler Principal Program Manager, Azure SDK
Light
Dark
0 comments