Bot to Human Handoff in Node.js

Lilian Kasem (she/her)

One common request from companies and organizations considering bots is the ability to “hand off” a customer from a bot to a human agent as seamlessly as possible. To solve this problem, we implemented an unopinionated e-2-e solution called Handoff, which enables bot authors to implement a wide variety of scenarios with minimal changes to the actual bot. We recently worked with a partner who wanted to use the Ibex dashboard to display all the Handoff interactions. A scenario we see often is a is help desk application; although little changes are needed to implement Handoff in your bot, creating your own help centre application that works with Handoff would require a bit of work.

This framework has been developed in TypeScript and Node.js, and a similar handoff scenario has also been developed in C# by Tomi Paananen. You can see his write-up of how the C# version works and his Intermediator-Bot sample on GitHub.

This project, Bot-Handoff, is open-sourced and available on GitHub. Contributions are always welcome!

How It Works

This framework connects Customers and Agents through the Microsoft Bot Framework.

The Handoff depends on a database of conversations, including a transcription of every message sent between the Customer and the bot, or between the Customer and the Agent.

A Handoff conversation consists of:

• address information for this conversation with the Customer
• the current state of this conversation (Bot, Waiting, or Agent)
• the conversational transcript
• address information for the Agent, if the Agent is currently connected to this Customer (Handoff does not record conversational metadata for the Agent, except when they are connected to a Customer)

Essentially, we have some middleware intercepting every message sent to and from the bot. If the message is from the bot, Customer, or an Agent who is speaking to a Customer, it transcribes the message before sending the message through.

If the message is from the Customer and contains the text “help me” (a command that can be configured by the author of the bot) we update the state of the Customer from ‘Bot’ to ‘Waiting’ meaning the user now waits for an Agent.

Once an Agent picks up the conversation, the state is changed from ‘Waiting’ to ‘Agent’. In this state, any message sent from the Customer is intercepted by the middleware and routed to the Agent using the stored address information for the Agent that picked up the conversation, and vice versa.

This code story will go into further detail about key aspects of this framework, including:

• Middleware Routing
• Transcripts and Logging
• Commands
• Storing State on MongoDB
• Application Insights
• Sentiment Analysis
• Force Handoff API
• Using the npm package in your bot
• Conclusion and resources

Middleware Routing

The heart of Handoff is the message router. Using the conversational metadata above, each message from a Customer, Bot, or Agent is routed appropriately. It is implemented as Bot middleware, and can be combined with any other middleware your bot is already using.

The code snippet below shows the implementation of  bot.use()  which is how middleware is implemented in the Node.js Microsoft Bot Framework SDK. The middleware performs two key actions: first, it checks to see if what the user enters matches any of our commands. For example, if the user is an Agent and types “list”, it will list all of the current conversations with the bot. Next, the middleware routes the message:

bot.use(
    commandsMiddleware(handoff),
    handoff.routingMiddleware(),
)

If the activity is from the Customer or Agent, it will go through the botbuilder  method. We only want to transcribe activities of type “message” so only in that scenario do we call routeMessage() ; otherwise, we just pass the activity through as usual. Activities from the bot go through the send method where we do something similar.

Here we also check to make sure there are no entities in the message activity. If there is an entity,  it means that the message is from the Agent acting as the bot. We don’t want the message to send twice as it will already be routed through the botbuilder  method. (When the Handoff is active, the Agent is speaking to the bot to speak to the Customer, and that same message is then routed as the bot.)

public routingMiddleware() {
    return {
        botbuilder: (session: builder.Session, next: Function) => {
            // Pass incoming messages to routing method
            if (session.message.type === 'message') {
                this.routeMessage(session, next);
            } else {
                // allow messages of non 'message' type through 
                next();
            }
        },
        send: async (event: builder.IMessage, next: Function) => {
            // Messages sent from the bot do not need to be routed
            // Not all messages from the bot are type message, we only want to record the actual messages  
            if (event.type === 'message' && !event.entities) {
                this.transcribeMessageFromBot(event as builder.IMessage, next);
            } else {
                //If not a message (text), just send to user without transcribing
                next();
            }
        }
    }
}

After getting a message activity from a Customer or Agent, we use routeMessage()  to check who the user is.

private routeMessage(session: builder.Session, next: Function) {
    if (this.isAgent(session)) {
        this.routeAgentMessage(session)
    } else {
        this.routeCustomerMessage(session, next);
    }
}

Agent Recognition

Customers and Agents are both just users connected to bots, so Handoff needs a way to identify an Agent. We use a function of the form isAgent(session: Session) => boolean  to determine if the user is an Agent, which can be customized by the author of the bot in the initial setup of the Handoff module. For example, in the snippet below, we determine that a user is an Agent if their username starts with “Agent”.

const isAgent = (session) => session.message.user.name.startsWith(“Agent”);

This information is passed by the bot author in the initial setup handoff.setup(bot, app, isAgent, { }) .

There are multiple ways this could be set up:

• Create a hardcoded directory of channel-specific user IDs for Agents (e.g., “Fred Doe on Facebook Messenger is one of our Agents”)
• Create a WebChat-based call center app that specially encodes Agent user IDs (e.g., “Agent001”, “Agent002”), which is easy with WebChat
• Create a WebChat-based call center app that authenticates users and then passes auth tokens to the bot via the WebChat backchannel
• Use authbot to identify the user as an Agent via OAuth2 (e.g., “This authenticated user is marked as an Agent in our employee database”)

Transcripts and Logging

Once we know who the user is, several checks are made before sending the message through the bot. In the Agent routing method, we check to see if the Agent is in a conversation; this means checking if they are currently talking to a Customer. If not, we don’t need further routing and we just pass the message through. Otherwise, if they are talking to a Customer, we route the message through the bot to the Customer they are in the conversation with.

const conversation = await this.getConversation({ AgentConversationId }, message.address);

// send text that Agent typed to the Customer they are in conversation with
this.bot.send(new builder.Message().address(conversation.Customer).text(message.text).addEntity({ "Agent": true }));

On the Customer side, we check their state. If they are taking to the bot, we just pass the message through. If they are waiting, we inform them they are being connected to an Agent. And finally, if they are in conversation with an Agent (and the Agent is in the conversation) we route the message through the bot to the Agent.

// this method will either return existing conversation or a newly created conversation if this is first time we've heard from Customer
const conversation = await this.getConversation({ CustomerConversationId: message.address.conversation.id }, message.address);

this.bot.send(new builder.Message().address(conversation.Agent).text(message.text));

For both scenarios, we update the conversation’s transcript array, using the conversation ID, so we have all the messages that have gone between the Customer and the bot, and the Customer and the Agent.

conversation.transcript.push({
    timestamp: datetime,
    from: from,
    sentimentScore: sentimentScore,
    state: conversation.state,
    text
});

The datetime variable used is of type new Date().toISOString() so that it matches the format returned from localTimestamp . Something worth noting is that localTimestamp  (a method returned with a message activity) is not provided when the message is coming from WebChat. The SDK recommends not using the timestamp  value,  so we put a check in place to use localTimestamp  where possible.

datetime = message.localTimestamp ? message.localTimestamp : message.timestamp

Commands

As mentioned earlier, the middleware of the bot looks for commands in all the messages sent to the bot. There are several commands made available through the commands.ts  middleware feature. After checking to make sure the activity is of the message type, we check if this command is coming from the Agent or from the Customer in a similar fashion as routeMessage() using isAgent .

On the Agent side, there are four commands available:

On the Customer side, there is only one command. The bot author can provide a custom keyword or phrase that a Customer can use to trigger the Handoff (the default is set to “help“) through an environment variable or the Handoff module initial setup options.

Regex is used to make sure the text is an exact match:

const customerStartHandoffCommandRegex = new RegExp("^" + indexExports._customerStartHandoffCommand + "$", "gi");

If the text matches exactly, the user’s message is transcribed, and they are queued to speak to an agent, changing their state from 0 (Bot) to 1 (Waiting).

Storing State on MongoDB

The conversation data, which consists of the items listed below, are all stored in a Mongo database:

• address information for this conversation with the Customer
• the current state of this conversation (Bot, Waiting, or Agent)
• the conversational transcript
• address information for the Agent, if the Agent is currently connected to this Customer (Handoff does not record conversational metadata for the Agent, except when they are connected to a Customer)

The Mongoose npm module was used to enable this feature; the bot author simply has to provide the MongoDB connection string as an option or environment variable. Another option is to use Azure Cosmos DB which automatically indexes all data and allows you to use the MongoDB API. You can learn more from this blog post on how to connect a MongoDB application to CosmosDB.

mongoose-provider.ts  handles all of the database interactions, and is where all of the schemas are described and, through a series of promises, the conversation is created, deleted, and updated.  For example:

export interface ConversationDocument extends Conversation, mongoose.Document { }

export const ConversationModel = mongoose.model<ConversationDocument>('Conversation', ConversationSchema)

private async updateConversation(conversation: Conversation): Promise<boolean> {
    return new Promise<boolean>((resolve, reject) => {
        ConversationModel.findByIdAndUpdate((conversation as any)._id, conversation).then((error) => {
            resolve(true)
        }).catch((error) => {
            console.log('Failed to update conversation');
            console.log(conversation as any);
            resolve(false);
        });
    });
}

Application Insights

The need to record all the conversation details in Azure Application Insights came from a hackfest we did with a partner. They wanted to use the Ibex dashboard to display all the Handoff interactions.

We implemented Application Insights as an optional feature by only logging the data to Application Insights if they provide the Application Insights instrumentation key in the initial setup of the handoff module, or through an environment variable.

We used the Application Insights npm module to set up and start an Application Insights client. This client is then used to track an event under the title “Transcript”.

let appInsights = require('applicationinsights');

appInsights.setup(_appInsightsInstrumentationKey).start();
exports._appInsights = appInsights;

The Application Insights logging happens in the addToTranscript()  method. Two things worth noting here:

1. You can’t log embedded JSON objects in Application Insights, so we flatten the object to one item.
2. We had to first stringify the object we got from MongoDB before parsing it so that functions from MongoDB don’t get logged JSON.parse(JSON.stringify(OBJECT)).

if (indexExports._appInsights) {   
    let latestTranscriptItem = conversation.transcript.length-1;
    let x = JSON.parse(JSON.stringify(conversation.transcript[latestTranscriptItem]));
    x['botId'] = conversation.Customer.bot.id;
    x['CustomerId'] = conversation.Customer.user.id;
    x['CustomerName'] = conversation.Customer.user.name;
    x['CustomerChannelId'] = conversation.Customer.channelId;
    x['CustomerConversationId'] = conversation.Customer.conversation.id;
    if (conversation.Agent) {
        x['AgentId'] = conversation.Agent.user.id;
        x['AgentName'] = conversation.Agent.user.name;
        x['AgentChannelId'] = conversation.Agent.channelId;
        x['AgentConversationId'] = conversation.Agent.conversation.id;
    }
    indexExports._appInsights.client.trackEvent("Transcript", x);    
}

Sentiment Analysis

We wanted to give bot authors the option to also log the sentiment score of messages from the Customer using Microsoft Cognitive Services Text Analytics; this API returns a value between 0 and 1 where 1 is a very positive sentiment and 0 is a very negative sentiment. If the user is not the Customer, or they do not provided an API key, the default sentiment score is set to -1.

let sentimentScore = -1;

...

if (from == "Customer") {
    if (indexExports._textAnalyticsKey) { 
        sentimentScore = await this.collectSentiment(text); 
    }
}

...

conversation.transcript.push({
    timestamp: datetime,
    from: from,
    sentimentScore: sentimentScore,
    state: conversation.state,
    text
});

We get the sentiment score by sending the user’s text to the Text Analytics API.

private async collectSentiment(text: string): Promise<number> {
    if (text == null || text == '') return;
    let _sentimentUrl = 'https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment';
    let _sentimentId = 'bot-analytics';
    let _sentimentKey = indexExports._textAnalyticsKey;

    let options = {
        url: _sentimentUrl,
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Ocp-Apim-Subscription-Key': _sentimentKey
        },
        json: true,
        body: {
            "documents": [
                {
                    "language": "en",
                    "id": _sentimentId,
                    "text": text
                }
            ]
        }
    };

    return new Promise<number>(function (resolve, reject) {
        request(options, (error, response, body) => {
            if (error) { reject(error); }
            let result: any = _.find(body.documents, { id: _sentimentId }) || {};
            let score = result.score || null;
            resolve(score);
        });
    });
}

Force Handoff via API

To provide additional means of controlling and customizing the Handoff experience, an endpoint (POST) is available to queue a Customer to speak to an Agent using the Customer’s conversation ID.  This API useful for a partner who wanted to use their own system to view all current conversations happening with a bot, and trigger the handoff manually with a button click, for any given conversation.

app.post('/api/conversations', async (req, res) => {
    const authHeader = req.headers['authorization'];
    if (authHeader) {
        if (authHeader === 'Bearer ' + _directLineSecret) {
            if (await handoff.queueCustomerForAgent({ customerConversationId: req.body.conversationId })) {
                res.status(200).send({ "code": 200, "message": "OK" });
            } else {
                res.status(400).send({ "code": 400, "message": "Can't find conversation ID" });
            }
        }
    } else {
        res.status(401).send({ "code": 401, "message": "Not Authorized" });
    }
});

You can use the GET endpoint to retrieve a list of all current conversations:

app.get('/api/conversations', async (req, res) => {
    const authHeader = req.headers['authorization'];
    console.log(authHeader);
    console.log(req.headers);
    if (authHeader) {
        if (authHeader === 'Bearer ' + _directLineSecret) {
            let conversations = await mongooseProvider.getCurrentConversations()
            res.status(200).send(conversations);
        }
    }
    res.status(401).send('Not Authorized');
});

The images below show a third party application took advantage of the Application Insights logs and this API:

Using the npm package

To make this framework easy to use, we packaged it up and made it available through npm under the name botbuilder-handoff.  Below is an example of how you can use the framework in a normal bot:

import * as builder from 'botbuilder';
import * as handoff from 'botbuilder-handoff';

//=========================================================
// Handoff Setup
//=========================================================

// Replace this function with custom login/verification for agents
const isAgent = (session: builder.Session) => session.message.user.name.startsWith("Agent");

/**
    bot: builder.UniversalBot
    app: express ( e.g. const app = express(); )
    isAgent: function to determine when agent is talking to the bot
    options: { }
**/
handoff.setup(bot, app, isAgent, {
    mongodbProvider: process.env.MONGODB_PROVIDER,
    directlineSecret: process.env.MICROSOFT_DIRECTLINE_SECRET,
    textAnalyticsKey: process.env.CG_SENTIMENT_KEY,
    appInsightsInstrumentationKey: process.env.APPINSIGHTS_INSTRUMENTATIONKEY,
    retainData: process.env.RETAIN_DATA,
    customerStartHandoffCommand: process.env.CUSTOMER_START_HANDOFF_COMMAND
});

The README for this project explains every option in detail.

Conclusion

This framework enables anyone developing a bot using the Node.js Bot Framework SDK to integrate bot to human handoff feature. The added functionality of Application Insights and the sentiment score provide bot authors with more data about interactions with their bot. This data can then be used to customize the bot-to-human handoff experience and analyze the overall customer experience with their bot and agents.

This project is open sourced and available on Github:Bot-Handoff. We continue to add major new features, but it is in a usable state now. Contributions to this project are always welcome!

Resources

• Bot-Handoff source code
• botbuilder-handoff npm module
• Ibex Dashboard
• Microsoft Bot Framework

Lilian Kasem (she/her) Senior Software Engineer, Commercial Software Engineering

Follow