{"id":4100,"date":"2018-02-05T10:45:17","date_gmt":"2018-02-05T18:45:17","guid":{"rendered":"https:\/\/www.microsoft.com\/reallifecode\/?p=4100"},"modified":"2020-03-14T17:42:03","modified_gmt":"2020-03-15T00:42:03","slug":"human-handoff-dashboard","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/human-handoff-dashboard\/","title":{"rendered":"Analysing Bot to Human Hand-off Interactions"},"content":{"rendered":"<p>Analytics is an important aspect of any application, whether it be a mobile app, a website, or even a bot. With the recent rise of bots, more and more engineers are interested in analyzing the interactions customers have with their bot. To address this growing demand our team built the <a href=\"https:\/\/github.com\/CatalystCode\/ibex-dashboard\">Ibex Dashboard<\/a>,\u00a0an <a href=\"https:\/\/azure.microsoft.com\/en-us\/services\/application-insights\/\">Application Insights<\/a>-based project that displays a bots analytics dashboard. We are currently expanding it to include other dashboard types, such as Health and Gaming.<\/p>\n<p>A common\u00a0use case in the bot-verse is the ability to hand over a customer conversation from a bot to a human, and the critical need for a feedback loop in this kind of scenario. During a recent engagement with\u00a0<a href=\"http:\/\/www.sage.com\/us\">Sage<\/a>, we developed a <a href=\"https:\/\/github.com\/CatalystCode\/ibex-dashboard\/blob\/master\/server\/dashboards\/preconfigured\/cosmosdb-handoff.ts\">Hand-off Dashboard Template<\/a> that can be used with the Ibex dashboard to provide details and insight about how a solution&#8217;s bot and agents are working with customers.<!--more--><\/p>\n<h2>Business Case<\/h2>\n<p>Recently we worked with <a href=\"http:\/\/www.sage.com\/us\">Sage<\/a>,\u00a0a software company that builds accounting products, to help\u00a0them implement the Ibex dashboard for their <a href=\"https:\/\/hellopegg.io\/?utm_campaign=SageOnePeggUSMicroSt&amp;utm_source=Sage-Websites&amp;utm_medium=NonPaid-OrganicSearch&amp;utm_content=SageOnePages&amp;utm_term=&amp;sFDCCampaignID=701240000002eh9\">Pegg<\/a> bot, which had already been developed with the <a href=\"https:\/\/dev.botframework.com\">Microsoft Bot Framework<\/a>.<\/p>\n<p>The Sage engineers also wanted to implement the bot to human hand-off feature with analytics about the hand-off interactions. They wanted to be able to answer important questions, such as:<\/p>\n<ul>\n<li>How many times has a customer\u00a0been handed over to a human?<\/li>\n<li>What are the average, longest and shortest waiting times for a customer who has been handed over to a human?<\/li>\n<li>How many people are currently waiting to speak to a human?<\/li>\n<li>How many people are currently talking to a human? How many to the bot?<\/li>\n<li>How many users does the bot currently have?<\/li>\n<li>How is the bot doing over time?<\/li>\n<li>Has there been an increase of people asking to speak to a human over time?<\/li>\n<li>What do the conversations between a customer and a bot and a customer and an agent look like?<\/li>\n<\/ul>\n<p>By analyzing conversation data, we are able to answer all of these questions. A dashboard that shows how your\u00a0bot is performing and how customers are interacting with it provides great feedback that can help you\u00a0improve the\u00a0bot. The expectation is that over time, the bot will handle more user needs, resulting in fewer switches to a human to continue the conversation.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2018\/02\/Screen-Shot-2017-06-30-at-11.43.49-AM.png\" alt=\"Image Screen Shot 2017 06 30 at 11 43 49 AM\" width=\"2552\" height=\"1000\" class=\"aligncenter size-full wp-image-10817\" srcset=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/Screen-Shot-2017-06-30-at-11.43.49-AM.png 2552w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/Screen-Shot-2017-06-30-at-11.43.49-AM-300x118.png 300w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/Screen-Shot-2017-06-30-at-11.43.49-AM-1024x401.png 1024w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/Screen-Shot-2017-06-30-at-11.43.49-AM-768x301.png 768w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/Screen-Shot-2017-06-30-at-11.43.49-AM-1536x602.png 1536w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/Screen-Shot-2017-06-30-at-11.43.49-AM-2048x803.png 2048w\" sizes=\"(max-width: 2552px) 100vw, 2552px\" \/><\/p>\n<h2>Bot to Human Hand-off\u00a0Node.js\u00a0Module<\/h2>\n<p>To get the hand-off dashboard working, we first had to implement the feature in the bot. A <a href=\"https:\/\/www.npmjs.com\/package\/botbuilder-handoff\">bot to human hand-off framework for the Microsoft Bot Framework Node.js SDK<\/a> had\u00a0already been developed. The framework, \u00a0<a href=\"https:\/\/www.npmjs.com\/package\/botbuilder-handoff\">botbuilder-handoff<\/a>, has been published to npm and\u00a0requires very little setup.<\/p>\n<pre class=\"lang:default decode:true \">const handoff = require('botbuilder-handoff');\r\n\r\n...\r\n\r\nhandoff.setup(bot, app, isAgent, {\r\n    mongodbProvider: config.MONGODB_PROVIDER,\r\n    directlineSecret: config.MICROSOFT_DIRECTLINE_SECRET,\r\n    textAnalyticsKey: config.CG_SENTIMENT_KEY,\r\n    appInsightsInstrumentationKey: config.APPINSIGHTS_INSTRUMENTATIONKEY,\r\n    retainData: config.RETAIN_DATA,\r\n    customerStartHandoffCommand: config.CUSTOMER_START_HANDOFF_COMMAND\r\n});<\/pre>\n<p>The Hand-off framework provided a great starting point, but there were a few additional features that the Sage team needed, detailed below.<\/p>\n<h3><strong>Azure Application<\/strong> Insights<strong>\u00a0Logging<\/strong><\/h3>\n<p>This feature enables engineers to have all logging data in one place, rather than having to pull data from several locations. The Hand-off framework originally only stored conversation data to MongoDB; we extended this capability by also logging the same data in Application Insights. We went with <a href=\"https:\/\/azure.microsoft.com\/en-us\/services\/application-insights\/\">Application Insights<\/a> because the Ibex dashboard currently uses it to display the general bot analytics.<\/p>\n<h3>Customer Message Sentiment Score<\/h3>\n<p>To improve the feedback loop about how the bot is doing with customers, we added a sentiment score for every message the user sends to the bot and to the human agent. This score can be used to analyze individual customer conversations and gauge customer satisfaction when interacting with the bot or an agent. It can also be used to automatically trigger a bot to human hand-off when the score goes below a certain threshold. We used the <a href=\"https:\/\/azure.microsoft.com\/en-us\/services\/cognitive-services\/text-analytics\/\">Microsoft Cognitive Services Text Analytics<\/a>\u00a0API for this feature.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2018\/02\/sample-angry-sentiment.png\" alt=\"Image sample angry sentiment\" width=\"670\" height=\"562\" class=\"aligncenter size-full wp-image-10816\" srcset=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/sample-angry-sentiment.png 670w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/sample-angry-sentiment-300x252.png 300w\" sizes=\"(max-width: 670px) 100vw, 670px\" \/><\/p>\n<p>You can read about how the latest version of the Hand-off framework was developed in our\u00a0<a href=\"\/developerblog\/2017\/06\/30\/bot-to-human-handover-in-node-js\/\">earlier code story<\/a>.<\/p>\n<h3>Message Notifications<\/h3>\n<p>When offering a bot service, it&#8217;s important to have the ability to notify the users about certain events, such as important messages or new bot features.<\/p>\n<p>Using the same structure as the Hand-off npm module, we developed a new npm package (<a href=\"https:\/\/www.npmjs.com\/package\/botbuilder-notifications\" rel=\"noopener\">botbuilder-notifications<\/a>)\u00a0that gives bot developers the ability to add an &#8220;admin to clients&#8221; notifications channel. With it, the admin user can schedule messages for its users and define the message content; when a user starts a new chat, the waiting messages will be displayed.<\/p>\n<p>The feature is consumed as a <a href=\"https:\/\/www.npmjs.com\/package\/botbuilder-notifications\">Node.js module<\/a>, and the integration process is straightforward:<\/p>\n<pre title=\"Integrating botbuilder-notifications into a bot\" class=\"lang:js decode:true\">const builder = require('botbuilder');\r\nconst notifications = require('botbuilder-notifications');\r\n\r\n...\r\n\r\nnotifications.setup(bot, app, isAgent, {\r\n    mongodbProvider: process.env.MONGODB_PROVIDER,\r\n    directlineSecret: process.env.MICROSOFT_DIRECTLINE_SECRET\r\n});<\/pre>\n<p>To use the feature, you must provide some sort of data store to save the messages. The current code base provides a default implementation that uses MongoDB, but could easily be modified to use other storage providers.<\/p>\n<h4><strong>Scenario Example<\/strong><\/h4>\n<p>To add a message to the notifications channel, the admin simply types &#8220;queue&#8221; followed by the message.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2018\/02\/queue-admin.jpg\" alt=\"Image queue admin\" width=\"1139\" height=\"712\" class=\"aligncenter size-full wp-image-10814\" srcset=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/queue-admin.jpg 1139w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/queue-admin-300x188.jpg 300w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/queue-admin-1024x640.jpg 1024w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/queue-admin-768x480.jpg 768w\" sizes=\"(max-width: 1139px) 100vw, 1139px\" \/><\/p>\n<p>Another user, who just started a new chat, will automatically get the notification:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2018\/02\/queue-user.jpg\" alt=\"Image queue user\" width=\"1783\" height=\"1199\" class=\"aligncenter size-full wp-image-10815\" srcset=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/queue-user.jpg 1783w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/queue-user-300x202.jpg 300w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/queue-user-1024x689.jpg 1024w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/queue-user-768x516.jpg 768w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/queue-user-1536x1033.jpg 1536w\" sizes=\"(max-width: 1783px) 100vw, 1783px\" \/><\/p>\n<p>The notifications\u00a0<a href=\"https:\/\/github.com\/CatalystCode\/bot-notifications\">source code<\/a> can be found on GitHub.<\/p>\n<h2>Capturing Bot Interactions Using Application Insights<\/h2>\n<p>To capture analytics for your bot when using Node.js,\u00a0 you can use the\u00a0<a href=\"https:\/\/github.com\/CatalystCode\/botbuilder-instrumentation\">botbuilder-instrumentation<\/a> package. There is also a simple instrumentation <a href=\"https:\/\/github.com\/CatalystCode\/botbuilder-instrumentation-cs\">package available for C#<\/a>\u00a0bot projects.<\/p>\n<pre class=\"lang:default decode:true\">npm install botbuilder-instrumentation \u2013-save\r\n<\/pre>\n<p>The basic logging setup is as follows:<\/p>\n<pre class=\"lang:js decode:true\">const instrumentation = require('botbuilder-instrumentation');\r\n\r\n\/\/ Setting up advanced instrumentation\r\nvar logging = new instrumentation.BotFrameworkInstrumentation({\r\n  instrumentationKey: process.env.APPINSIGHTS_INSTRUMENTATIONKEY,\r\n  sentiments: {\r\n    key: process.env.CG_SENTIMENT_KEY,\u00a0 \r\n  }\r\n});\r\n\r\nlogging.monitor(bot);\r\n<\/pre>\n<p><span class=\"lang:default decode:true crayon-inline \">APPINSIGHTS_INSTRUMENTATIONKEY<\/span>\u00a0 is your <a href=\"http:\/\/bit.ly\/AppInsights\">Application Insights<\/a> <strong>Instrumentation key<\/strong> and <span class=\"lang:default decode:true crayon-inline \">CG_SENTIMENT_KEY<\/span>\u00a0 is your <a href=\"https:\/\/www.microsoft.com\/cognitive-services\/en-us\/text-analytics-api\">Microsoft Cognitive Services<\/a>\u00a0<strong>Sentiment Analytics key<\/strong>.<\/p>\n<p><strong>Tip:<\/strong> for running on localhost, you can use the <a href=\"https:\/\/www.npmjs.com\/package\/dotenv\" rel=\"noopener\">dotenv npm package<\/a>\u00a0to manage all the various environment variables.<\/p>\n<pre class=\"lang:js decode:true\">\"env\": {\r\n  \"APPINSIGHTS_INSTRUMENTATIONKEY\": \"YOUR_APP_INSIGHTS_INSTRUMENTATION_KEY\",\r\n  \"CG_SENTIMENT_KEY\": \"YOUR_LUIS_TEXT_ANALYTICS_KEY\"\r\n}\r\n<\/pre>\n<p>For more information, refer to the <a href=\"https:\/\/github.com\/CatalystCode\/botbuilder-instrumentation\/blob\/master\/README.md\">instrumentation readme<\/a> section about capturing custom fields and events.<\/p>\n<p>Once your bot instrumentation is set up, the bot data will start logging into Application Insights. You can try writing some queries in the Application Insights online query editor (using autocomplete). Please note it may take 5 minutes or so before your data is ready to query.<\/p>\n<pre class=\"lang:default decode:true\">customEvents\r\n| where timestamp &gt; ago(1d)\r\n<\/pre>\n<p>In this instance, <code>customEvents<\/code> is the table name and <code>&gt; ago(1d)<\/code> fetches records that are no more than a day old.<\/p>\n<p>The Application Insights query language is well suited for visual illustration purposes.\u00a0 Application Insights allows users to query using multiple <code>where<\/code> clauses and <code>summarize<\/code> statements to aggregate the data as required. The <code>bin()<\/code> function is very useful for grouping sets of data sharing the same date or time interval. For more information, please refer to the <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/application-insights\/app-insights-analytics\">online documentation<\/a>.<\/p>\n<h2>Creating a &#8216;hand-off&#8217; dashboard for the bot<\/h2>\n<p>Once you have captured bot data in Application Insights using <a href=\"https:\/\/github.com\/CatalystCode\/botbuilder-instrumentation\/blob\/master\/README.md\">instrumentation<\/a>, you can create your own visualizations using the <a href=\"https:\/\/github.com\/CatalystCode\/ibex-dashboard\">Ibex dashboard<\/a> open source project. You can either choose one of the built-in templates and paste in your keys or add your own queries. The <a href=\"https:\/\/github.com\/CatalystCode\/ibex-dashboard\/blob\/master\/server\/dashboards\/preconfigured\/cosmosdb-handoff.ts\">bot &#8216;hand-off&#8217; template<\/a> was created using the following Application Insights queries (using test data as an illustration).<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2018\/02\/AppInsights-Online-Query-Editor.png\" alt=\"Image AppInsights Online Query Editor\" width=\"1207\" height=\"443\" class=\"aligncenter size-full wp-image-10811\" srcset=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/AppInsights-Online-Query-Editor.png 1207w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/AppInsights-Online-Query-Editor-300x110.png 300w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/AppInsights-Online-Query-Editor-1024x376.png 1024w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/AppInsights-Online-Query-Editor-768x282.png 768w\" sizes=\"(max-width: 1207px) 100vw, 1207px\" \/><\/p>\n<h4>Total users currently speaking with bot or human or awaiting hand-off<\/h4>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2018\/02\/ibex-scorecard-users.png\" alt=\"Image ibex scorecard users\" width=\"629\" height=\"119\" class=\"aligncenter size-full wp-image-10813\" srcset=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/ibex-scorecard-users.png 629w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2018\/02\/ibex-scorecard-users-300x57.png 300w\" sizes=\"(max-width: 629px) 100vw, 629px\" \/><\/p>\n<p>The following query displays a scorecard overview of total users and their hand-off state:<\/p>\n<pre class=\"lang:default decode:true\">customEvents \r\n| where timestamp &gt; ago(90d) \r\n| where name == 'Transcript' \r\n| extend customerId=tostring(customDimensions.customerId) \r\n| extend state=toint(customDimensions.state) \r\n| extend timestamp=todatetime(customDimensions.timestamp) \r\n| project customerId, timestamp, state\r\n| order by timestamp desc\r\n| summarize transcripts_count=count(customerId), timestamps=makelist(timestamp) by customerId, state \r\n| project customerId, state, transcripts_count, timestamp=timestamps[0]\r\n| summarize count(customerId), totals=makelist(transcripts_count), states=makelist(state), timestamps=makelist(timestamp) by customerId\r\n| project customerId, state=toint(states[0]), transcripts_count=toint(totals[0]), timestamp=timestamps[0]Note that there multiple aggregations using the <code style=\"font-size: 16px\">summarize<\/code><span style=\"background-color: #ffffff;font-family: Georgia, 'Times New Roman', 'Bitstream Charter', Times, serif;font-size: 16px\"> statement.<\/span><\/pre>\n<p>The first <em>summarize<\/em> statement groups a user&#8217;s transcripts by state.\u00a0 The second <em>summarize<\/em> statement groups the results by user ID,\u00a0 so we can show how many users are talking to the bot, talking to a human agent, or waiting for a human agent to respond.<\/p>\n<h4>Time waiting for human hand-off<\/h4>\n<p><!-- TODO - missing image --> \n<!-- \n\n<p><img decoding=\"async\" class=\"alignnone wp-image-5279 size-full\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/ibex-scorecard-times.png\" alt=\"\" width=\"629\" height=\"119\" \/><\/p>\n\n--><\/p>\n<p>To find out the time waiting for human hand-off, we calculate the time difference between transcripts of a user when the state changes from waiting for a\u00a0human agent to talking with a\u00a0human agent. With <strong>botbuilder<\/strong>,\u00a0the local timestamp fields are expressed as <a href=\"https:\/\/docs.botframework.com\/en-us\/node\/builder\/chat-reference\/classes\/_botbuilder_d_.message.html#localtimestamp\" rel=\"noopener noreferrer\">ISO string dates<\/a> which can be parsed in Application Insights using the <em>todatetime()<\/em> function.<\/p>\n<pre class=\"lang:default decode:true\">customEvents \r\n| where timestamp &gt; ago(90d) \r\n| where name == 'Transcript'\r\n| extend conversationId=tostring(customDimensions.userConversationId), customerId=tostring(customDimensions.customerId), state=toint(customDimensions.state)  \r\n| where state==1 or state==2\r\n| order by timestamp asc\r\n| summarize total=count(), times=makelist(timestamp) by conversationId, customerId, bin(state, 1)\r\n| project conversationId, customerId, state, startTime=times[0] \r\n| summarize result=count(state), startEndTimes=makelist(startTime) by conversationId, customerId\r\n| where result == 2\r\n| project conversationId, customerId, timeTaken=todatetime(startEndTimes[1])-todatetime(startEndTimes[0])The initial part of the query lists the customer id and state.<\/pre>\n<p>The first <em>summarize<\/em> statement reduces the list to show state change (of value of 1, where 0 is bot, 1 is waiting for an agent and 2 is agent) with total transcripts with the timestamp. The second <em>summarize<\/em> statement groups the state changes by user so that the difference in time can be calculated as time taken using the projection function.<\/p>\n<p>There is an advantage of doing these aggregations using Log Analytics query language rather than coding this logic yourself in JavaScript. If you want to learn more, see the related <a href=\"http:\/\/www.deadlyfingers.net\/azure\/querying-application-insights-for-data-visualisation\/\">developer blog post about the Ibex Dashboard<\/a>.<\/p>\n<p>From these results, we can work out the fastest, longest and average times waiting in the template <em><a href=\"https:\/\/github.com\/CatalystCode\/ibex-dashboard\/blob\/master\/server\/dashboards\/preconfigured\/cosmosdb-handoff.ts#L160\" rel=\"noopener noreferrer\">calculated<\/a><\/em>\u00a0method.<\/p>\n<pre class=\"lang:js decode:true\">calculated: (results) =&gt; {\r\n  const times = results.reduce((acc, cur) =&gt; {\r\n    \/\/ converts time hh:mm:ss format to value in seconds\r\n    acc.push(cur.timeTaken.split(':').reverse().reduce((a, c, i) =&gt; a + c * Math.pow(60, i), 0));\r\n    return acc;\r\n  }, []);\r\n  const avgTimeWaiting = times.reduce((a, c) =&gt; a + c, 0) \/ times.length;\r\n  const maxTimeWaiting = Math.max(...times);\r\n  const minTimeWaiting = Math.min(...times);\r\n  return {\r\n    'transcriptsTimeWaiting-avg': isFinite(avgTimeWaiting) ? avgTimeWaiting : '-',\r\n    'transcriptsTimeWaiting-longest': isFinite(avgTimeWaiting) ? maxTimeWaiting : '-',\r\n    'transcriptsTimeWaiting-shortest': isFinite(avgTimeWaiting) ? minTimeWaiting : '-',\r\n  };\r\n}\r\n<\/pre>\n<h4>Scorecard and timeline of total conversations with bot\/human<\/h4>\n<p><!-- TODO - missing image -->\n<!-- \n\n<p><img decoding=\"async\" class=\"alignnone wp-image-5278\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/ibex-scorecard-messages-196x300.png\" alt=\"\" width=\"157\" height=\"240\" \/><img decoding=\"async\" class=\"alignnone wp-image-5277 size-large\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/ibex-timeline.png\" alt=\"\" width=\"780\" height=\"240\" \/><\/p>\n\n--><\/p>\n<p>This <span class=\"annotation\" data-author=\"Clemens Wolff\" data-annotation=\"Would it make sense to talk about the business use-case for these queries here? E.g. what business concern do the queries\/visualizations represent? We may be able to leverage the questions\/concerns that Sage had during the Ibex hackfest for these details.\">query <\/span>is used to display both a scorecard and timeline for total messages sent based on hand-off state. The timeline is one of the most telling visuals, as it shows how well the QnA knowledge performs over time. As the knowledge base improves, the requirement for human hand-off should diminish.<\/p>\n<pre class=\"lang:default decode:true\">customEvents \r\n| where timestamp &gt; ago(90d) \r\n| where name == 'Transcript' \r\n| extend customerName=tostring(customDimensions.customerName), text=tostring(customDimensions.text), state=toint(customDimensions.state), agentName=tostring(customDimensions.agentName), from=tostring(customDimensions.from) \r\n| extend timestamp=todatetime(customDimensions.timestamp) \r\n| extend states=pack_array('bot','waiting','agent','watching')\r\n| extend stateLabel=tostring(states[state])\r\n| where state == 0 or state == 2\r\n| project timestamp, from, text, customerName, agentName, state, stateLabel \r\n| summarize transcripts_count=count() by bin(timestamp, 1d), state, stateLabel \r\n| order by timestamp ascThis query returns list of users transcripts with state labels ordered by timestamp.<\/pre>\n<p>The <em>summarize<\/em> statement reduces the list to count the number of transcripts given a time interval. The time interval is used for charting the total transcripts with the bot vs. an agent on the timeline.<\/p>\n<h4>List recent conversations with sentiment<\/h4>\n<p><!-- TODO - missing image -->\n<!-- \n\n<p><img decoding=\"async\" class=\"alignnone wp-image-5281 size-large\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/ibex-data-table.png\" alt=\"\" width=\"780\" height=\"148\" \/><\/p>\n\n --><\/p>\n<p>This query adds conversations and sentiment to the data table. Here the operator might wish to review recent conversations with negative sentiment and initiate a hand-off to human.<\/p>\n<pre class=\"lang:default decode:true\">customEvents \r\n| where timestamp &gt; ago(90d) \r\n| where name == 'Transcript'\r\n| extend conversationId=tostring(customDimensions.userConversationId), customerName=tostring(customDimensions.customerName), timestamp=todatetime(customDimensions.timestamp) \r\n| project conversationId, customerName, timestamp, customDimensions\r\n| order by timestamp desc\r\n| summarize transcripts_count=count(timestamp), transcripts=makelist(customDimensions) by customerName, conversationId\r\n| project conversationId, customerName, transcripts_count, transcriptsThe initial part of the query lists the customer transcripts ordered by timestamp.<\/pre>\n<p>The <em>summarize<\/em> statement groups the transcripts by user\u00a0so we can display the last message and sentiment as well as current status (such as whether they are currently talking to a bot or human agent).<\/p>\n<h2>Conclusion<\/h2>\n<p>The ability to analyze bot-to-human transitions and customer sentiment makes the Ibex Dashboard even more useful to engineers who want to continue to refine and improve their bots&#8217; performance. The addition template we created for the Ibex Dashboard allows you to analyse the interactions your customers\u00a0are having with your bot, as well as with agents. It can also help you to improve your bot by showing where your users are getting frustrated and where an agent has had to step in.<\/p>\n<p>Through the reuse of the bot to human hand-off npm, we also developed a simple npm package to provide a\u00a0notification feature for any bot developed with the Microsoft Bot Framework. This feature is useful when you want to be able to push new information to your customers.<\/p>\n<h3>Future Work and Reuse<\/h3>\n<p>The <a href=\"https:\/\/github.com\/CatalystCode\/ibex-dashboard\" rel=\"noopener noreferrer\">Ibex dashboard<\/a> comes with a set of <a href=\"https:\/\/github.com\/CatalystCode\/ibex-dashboard\/tree\/master\/client\/src\/components\/generic\" rel=\"noopener noreferrer\">generic elements<\/a> that can be adapted to many different scenarios. To take this project further, you could fork it and add your own generic elements and template queries to the Ibex dashboard. For more information on the dashboard and using generic components, please check the <a href=\"https:\/\/github.com\/CatalystCode\/ibex-dashboard\/tree\/master\/docs\" rel=\"noopener noreferrer\">readme docs<\/a>\u00a0included in the repo.<\/p>\n<p>You can find the project <a href=\"https:\/\/github.com\/palindromed\/Bot-HandOff\/tree\/npm-handoff\">source code on GitHub<\/a>; we welcome contributions!<\/p>\n<h3><\/h3>\n","protected":false},"excerpt":{"rendered":"<p>A common use case in the bot-verse is the ability to hand over a customer conversation from a bot to a human, and the vital need for a feedback loop for this kind of scenario. From this, we developed a Hand-off Dashboard Template that can be used within the Ibex dashboard to provide users with important information about how their bot and agents are working with customers.<\/p>\n","protected":false},"author":21393,"featured_media":10812,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[13],"tags":[64,249],"class_list":["post-4100","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-bots","tag-azure-application-insights","tag-microsoft-bot-framework-mbf"],"acf":[],"blog_post_summary":"<p>A common use case in the bot-verse is the ability to hand over a customer conversation from a bot to a human, and the vital need for a feedback loop for this kind of scenario. From this, we developed a Hand-off Dashboard Template that can be used within the Ibex dashboard to provide users with important information about how their bot and agents are working with customers.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/4100","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/21393"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=4100"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/4100\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media\/10812"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=4100"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=4100"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=4100"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}