We’re excited to announce the public preview of the Microsoft AI Chat Protocol library for building AI-powered frontends in JavaScript. The library and corresponding API specification are available on GitHub.

AI stands at the forefront of the tech industry’s latest innovations, causing a rapid spike in the number of technologies available for building AI applications. These technologies range from different Large Language Models (LLMs), orchestration frameworks, evaluation techniques, and design patterns for AI infrastructure such as Retrieval-Augmented Generation (RAG). While these solutions have their place in building a robust AI backend to serve users and handle complex tasks, we noticed that on the frontend, developers don’t get the same level of guidance or tooling when building applications.

This diagram details these two separate workstreams for developers. It’s important to note that this diagram is intended to be a high-level example of how AI chat applications are designed today. Any implementation-specific details are intentionally abstracted away to clearly identify the two different development workstreams.

One side is focused on client-side AI service consumption on the frontend in the browser, and the other is focused on the AI backend. In between the frontend and AI backend, there’s a “middle tier” endpoint that the frontend can call to access the AI backend. The diagram uses an Azure OpenAI Service-hosted model with LangChain orchestration as an example of what an AI backend can look like.

If this middle-tier endpoint adheres to the AI Chat Protocol API specification, the frontend can use the library to get all the benefits you might be familiar with in the Azure SDK client libraries via Azure Core. These benefits include authentication, retries, HTTP pipeline management, and most importantly for this library, streaming responses, which can update your interfaces in real time. The consistent API surface enables the AI backend to be composed of different models and orchestration tools.

Importantly, the AI Chat Protocol API specification allows for standardization of the AI backend consumption process. That means as long as the middle-tier endpoint adheres to this specification, you can consume any AI backend in a consistent and robust way. The specification allows for flexibility around the models, orchestration tools, and architectural design patterns used on the AI backend. Additionally, this API standardization allows for a consistent way of performing evaluations on different AI backends due to a unified consumption pattern. With consistent consumption and evaluations, developers can work more on building applications and fine-tuning their AI backends for customer value instead of focusing heavily on their application architecture.

Library Fundamentals

As mentioned before, because the library is built on a similar foundation as Azure Core, the process of using the library should feel familiar to Azure SDK customers.

Installation

The AI Chat Protocol library is available on npm for you to download.

npm i @microsoft/ai-chat-protocol

Usage

Once you have the library installed, you can start by creating a client object, just like other Azure SDK libraries. Next, we pass in a required endpoint when instantiating the client. This endpoint should adhere to the AI Chat Protocol API specification. If you’re using any authentication, you can also pass in your TokenCredential here.

import {
    AIChatMessage,
    AIChatProtocolClient,
} from "@microsoft/ai-chat-protocol";

const client = new AIChatProtocolClient("/api/chat");

Once you have a client in your frontend application code, you can then send a message to your middle-tier endpoint and receive output via either the getCompletion method for a synchronous response, or via the getStreamedCompletion method for a streamed response to your frontend. When using the getStreamedCompletion method, the response can include information about the role of the sender, the session state, or the content of the message. This example uses React Hooks for handling sessionState, which is an unknown type passed in when calling getStreamedCompletion in order to give the model an updated state of the conversation.

const [sessionState, setSessionState] = useState<unknown>(undefined);

const message: AIChatMessage = {
    role: "user",
    content: "Hello World!",
};

const result = await client.getStreamedCompletion([message], {
    sessionState: sessionState,
});

for await (const response of result) {
    if (response.sessionState) {
        //do something with the session state returned
    }
    if (response.delta.role) {
        // do something with the information about the role
    }
    if (response.delta.content) { 
        // do something with the content of the message
    }
}

If your backend adheres to the AI Chat Protocol API specification, this starter code is all you need to get started with the AI Chat Protocol library.

Summary

The Microsoft AI Chat Protocol library for JavaScript/TypeScript provides a way for developers to consistently and easily stream AI backend responses to their applications. The AI Chat Protocol API specification creates a consistent surface for AI backend consumption and evaluations, letting developers focus on providing value in their applications instead of getting their endpoints to play nicely with one another. The AI Chat Protocol library for JavaScript/TypeScript is available now on npm and is open-source on GitHub.

Since the library is in public preview, any feedback would be appreciated. End-to-end samples are available on Azure and are located in the GitHub repo.

• AI Chat Protocol Library GitHub
• AI Chat Protocol API Specification

If you’re interested in watching the AI Chat Protocol library session from Microsoft Build, check out this video!