Unit Testing for Bot Applications

Mor Shemesh

Image: Mr Robot has some RAM by Chris Isherwood, used by CC BY 2.0

First Things First

Writing code using ad-hoc technologies like Microsoft Bot Framework (MBF) is fun and exciting. But before rushing to code bots that can make coffee and send spaceships to the moon, you need to think about unit testing your code.

Recently, we worked with Moed.ai on a bot that schedules tasks and manages resources’ time slots. In order to ensure the quality of their bot Moed.ai wanted the ability to unit-test their bot logic.

This code story outlines the way we tackled the challenge of adding unit test support in Microsoft Bot Framework.

What Should We Test?

The first question we tried answering was “What are we trying to test?”

Bots built using MBF dialogs and intents usually consist of code intertwined with framework code. Moreover, major parts of the framework require external connectivity for saving session states or querying LUIS models.

In most projects, when we think about unit testing, we think about segregated sections of logic that receive input and are expected to return specific output. In case this logic has external dependencies, we need to search for ways to abstract those dependencies and provide stubs during testing.

This means we can take one of the following approaches:

1. Move the project’s code to a separate module or directory and remove any dependencies on MBF
2. Find a way to test the conversational logic of the bot without external dependencies

Code Separation Difficulty

Writing code for bots is a little different than writing code for other web-based applications.

Writing code for bot applications looks a lot like describing the way a conversational flow may evolve. For example, just like in a customer service conversation, the bot application waits for a request that it can understand from the user to start a meaningful conversation. Then, during a series of questions and answers, it collects more data while trying to understand and provide the user with the best service. Taking maintainability and readability into account, it is important that such conversations represented by your code are understandable.

Moreover, in bot scenarios, many actions and decisions require a high level of dependency on MBF.

Here are some examples of code that are dependent on MBF:

// Prompting the user for input
builder.Prompts.choice(session, 'What would you like to do?', [ "Play Games", "Do serious work", "Other" ]);

// Saving data in session variable
session.conversationData.userChoice = "Play games while appearing to work seriously";

// Beginning new dialogs or sending a "typing" status
session.send("Give a raise to smart worker");
session.sendTyping();

// Ending a conversation
session.endDialog("Good Bye");

A conversational flow consists of multiple REST calls and replies between the user and the web server. Trying to separate our logic from the conversational flow in many cases left us with almost no code to test. Moreover, that separation left the conversational flow logic untested.

When we thought about unit testing in the world of bots, we realized that conversational flow logic is intertwined with the core logic of our application. This realization led us to develop a unique approach to testing the bot.

How Do We Test Conversational Flow Logic?

For the purpose of this case study, we create a sample Alarm Clock Bot. A working copy of this bot application is available in this GitHub repo.

Let’s look at the following breakdown of a request to the bot application:

• Receives request from API
• Directs request to MBF
• Performs dialog step with bot input
• Communicates back to MBF

The code we want to test is wedged in the middle of this flow, but is entirely orchestrated by MBF.

Microsoft Bot Framework offers a class called ConsoleConnector. This class enables simulation of communications with the bot object without requiring an external connection:

var builder = require('botbuilder');

var connector = new builder.ConsoleConnector();

// BotToTest is the bot class exposing a collection of intents.
// To understand how to build such a class, follow the code in the Alarm Clock Bot repository
var bot = new BotToTest(connector);
bot.on('send', function (message) {
  /* Check returned message */
});

connector.processMessage('Hello World');

Next, let’s see how to test multiple steps in a dialog.

For this purpose we use a step indicator to tell us the index of the message:

var step = 1;
bot.on('send', function (message) {
  
  switch(step++) {
    case 1:
      assert(message.text == 'What would you like to do?');
      connector.processMessage('Play Games');
      break;

    case 2:
      assert(message.text == 'When?');
      connector.processMessage('Always');
      break;

    case 3:
      assert(message.text == 'Why?');
      connector.processMessage('I'm too cool for school');
      break;

    case 4:
      assert(message.text == 'No problem');
      break;

    default:
      assert(false); // The conversation should have ended
  }
  
});
connector.processMessage('Hello World');

To enable easy addition of more tests to the suite, we moved the conversational flow to an external module that exports JSON and consumes that via a generic tester:

/test/dialog-flows/context-switching.js:

module.exports = [
  {
    out: "set alarm in 10 seconds"
  },
  {
    in: "What would you like to call your alarm?",
    out: "test"
  },
  {
    in: /^(Creating alarm named "test" for)/i,
    out: "delete alarm named test"
  },
  {
    in: "Deleted the 'test' alarm." // the message sent by the bot after a few seconds
  }
];

/test/common.js

function testBot(bot, messages, done) {
  var step = 1;
  var connector = bot.connector();
  bot.on('send', function (message) {
      
    var check = messages[step - 1];
    
    // Check input message
    if (check.in) {
      assert(message.text === check.in);
    }

    // Send an output reply
    if (check.out) {
      connector.processMessage(check.out);
    }

    // End conversation in the last message
    step++;
    if (step - 1 == messages.length) {
      done();
    }
  });
}

module.exports = {
  testBot
}

Testing LUIS

LUIS is an external service integrated into the MBF SDK and used by many bot applications for intent recognition and entity extraction. To enable testing of that service we need to mimic URL calls to LUIS. We used Nock to simulate LUIS’s call, but Nock can be used to mimic any URL calls.

For the following mocked-up call, we made a GET call to LUIS with the appropriate query and copied the responses:

  nock('https://luis.url')
    .get('/?id=appId&subscription-key=subId&q=' + encodeURIComponent('set alarm test in 10 seconds'))
    .reply(200, {
      "query": "set alarm test in 10 seconds",
      "intents": [
        {
          "intent": "builtin.intent.alarm.set_alarm"
        }
        /* ... */
      ],
      "entities": [
        {
          "entity": "in 10 seconds",
          "type": "builtin.alarm.start_time",
          "resolution": {
            "resolution_type": "builtin.datetime.time",
            "time": "2016-12-14T15:31:59" // some time in the past
          }
        }
      ]
    })
    .get('/?id=appId&subscription-key=subId&q=' + encodeURIComponent('delete alarm named test'))
    .reply(200, {
      "query": "delete alarm named test",
      "intents": [
        {
          "intent": "builtin.intent.alarm.delete_alarm"
        }
      ],
      "entities": [
        {
          "entity": "test",
          "type": "builtin.alarm.title"
        }
      ]
    });

Creating The Suite

Finally, to wrap it all up in a test suite, we used a describe call:

/* requires */

var historyMessages = require('./dialog-flows/history-intents');
var switchingMessages = require('./dialog-flows/context-switch');
common.setup();

//Our parent block
describe('Bot Tests', () => {

  it('should recognize history intents', function (done) { 
      var connector = new builder.ConsoleConnector();
      var bot = TestedBot.create(connector);

      common.testBot(bot, historyMessages, done);
      
      connector.processMessage('hi');
  });

  it('context switching', function (done) { 
      var connector = new builder.ConsoleConnector();

      var bot = TestedBot.create(connector);       
      common.testBot(bot, switchingMessages, done);
      
      connector.processMessage('hi');
  });
});

To be able to do that, we needed the bot to expose a create call that receives a connector object:

var builder = require('botbuilder');

function create(connector) {

  var bot = new builder.UniversalBot(connector);
  var intents = new builder.IntentDialog();

  bot.dialog('/', intents);
  /* ... */

  return bot;
}

module.exports = { create };

This way, we can supply a builder.ChatConnector in runtime and a builder.ConsoleConnector in testing.

Opportunities for Reuse

You should reuse this code in any Microsoft Bot Framework project that requires unit testing for conversational flow.

Repositories

This is a sample of building a Set Alarm project with integrated unit testing: https://github.com/CatalystCode/alarm-bot-unit-testing It can also be used as a quickstart template for developing bots.

To see the full solution integrated into a bot see: https://github.com/CatalystCode/multilingual-uber-bot

Mor Shemesh

Follow