{"id":2124,"date":"2016-12-10T02:08:04","date_gmt":"2016-12-10T10:08:04","guid":{"rendered":"https:\/\/www.microsoft.com\/reallifecode\/index.php\/2016\/12\/10\/leveraging-azure-search-for-implementing-a-qna-bot-in-unsupported-languages\/"},"modified":"2020-03-17T23:52:19","modified_gmt":"2020-03-18T06:52:19","slug":"leveraging-azure-search-for-implementing-a-qna-bot-in-unsupported-languages","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/leveraging-azure-search-for-implementing-a-qna-bot-in-unsupported-languages\/","title":{"rendered":"Leveraging Azure Search for Implementing a QnA Bot in Unsupported Languages"},"content":{"rendered":"<p><em>Image: <a href=\"https:\/\/www.flickr.com\/photos\/42931449@N07\/5397530925\">F.A.Q<\/a> by <a href=\"https:\/\/www.flickr.com\/photos\/42931449@N07\">photosteve101<\/a>, used by <a href=\"https:\/\/creativecommons.org\/licenses\/by\/2.0\">CC BY 2.0<\/a><\/em><\/p>\n<h2 id=\"background\">Background<\/h2>\n<p>Our team recently worked with a Korean company that wanted to expose a QnA (<em>Frequently Asked Questions<\/em> \/ <em>Questions and Answers<\/em>) bot in their web portal to help answer common user questions. Bot Framework provides a QnA service (<a href=\"https:\/\/qnamaker.botframework.com\">QnA Maker<\/a>) that takes a list of common questions and answers related to your domain, called a QnA corpus, as input. After indexing the QnA corpus, the QnA Maker provides a simple REST API service to get the most suitable answer for a free text-based question.<\/p>\n<h2 id=\"the-problem\">The Problem<\/h2>\n<p>Since the QnA Maker currently does not support Korean, we had to develop similar QnA functionality without using the QnA Maker service.<\/p>\n<h2 id=\"the-solution\">The Solution<\/h2>\n<p>This post describes a method for providing a backend service for QnA services in languages that the QnA Maker does not currently support, using Azure Search. For the purpose of demonstration, we will build a QnA service in the English language. However, you can apply the following steps to any language currently supported by Azure Search (please refer to the <a href=\"https:\/\/msdn.microsoft.com\/en-gb\/library\/azure\/dn879793.aspx\">list of supported languages<\/a>).<\/p>\n<p><a href=\"https:\/\/azure.microsoft.com\/en-us\/services\/search\/\">Azure Search<\/a> is a <em>search engine as a service<\/em> offered by Azure. Azure Search can be used to index custom data and run search queries on the indexed corpus. Using Azure Search, we can index our QnA items and find the most appropriate item and label based on the question that the user asked. In addition to supporting most languages, the following two features make Azure Search the right tool for building a custom QnA service:<\/p>\n<ol>\n<li><strong>Analyzers<\/strong>&#8211; an analyzer augments the search query with a dictionary of synonyms and related phrases to increase the likelihood of finding a related match in the corpus.<\/li>\n<li><strong>Scoring Profile<\/strong>&#8211; a scoring profile allows us to provide larger or smaller weights to the different fields of the indexed items. This profile is used by Azure Search engine in the scoring process and impacts the results directly as a function of the scoring profile.<\/li>\n<\/ol>\n<p>In our solution we will provide Azure Search with our QnA corpus, in the form of a table containing the following schema:\n[<code class=\"highlighter-rouge\">id<\/code>, <code class=\"highlighter-rouge\">category<\/code>, <code class=\"highlighter-rouge\">url<\/code>, <code class=\"highlighter-rouge\">question<\/code>, <code class=\"highlighter-rouge\">answer<\/code>, <code class=\"highlighter-rouge\">keywords<\/code>]<\/p>\n<ul>\n<li><code class=\"highlighter-rouge\">id<\/code> is the unique key for each item<\/li>\n<li><code class=\"highlighter-rouge\">keyword<\/code> is a collection of keywords that best describes the question<\/li>\n<li><code class=\"highlighter-rouge\">url<\/code> is for cases where we want to direct the user to an external URL as part of the answer<\/li>\n<li><code class=\"highlighter-rouge\">category<\/code> will be used and explained later in a more advanced scenario<\/li>\n<\/ul>\n<p><em>A <a href=\"https:\/\/dxcodestories.westus.cloudapp.azure.com\/reallifecode\/wp-content\/uploads\/2016-12-10-Azure-Search-QNA\/AzureSearchQnA.xlsx\">sample excel file<\/a> is available<\/em><\/p>\n<p>Using the scoring profile feature, we can define a few scoring profiles and then try each to see which provides better results. There are no magic numbers here and we\u2019ll have to experiment with different weights so that we get the most accurate results.<\/p>\n<p>Our solution is based on the keyword field. We use this field to <em>promote<\/em> the essence of the question text into this field. By providing this field with the highest weight, we are \u201cpushing\u201d the results in that direction, making the items with the most similar keywords to the keyword field appear with a higher score.<\/p>\n<p>In this demo, I\u2019ll create a scoring profile that defines the following numerical weights for the fields as explained above:<\/p>\n<ul>\n<li><code class=\"highlighter-rouge\">keywords<\/code> &#8211; 5<\/li>\n<li><code class=\"highlighter-rouge\">question<\/code>&#8211; 3<\/li>\n<li><code class=\"highlighter-rouge\">answer<\/code>&#8211; 2<\/li>\n<\/ul>\n<p>This solution assumes that the keywords field contains the most relevant words that the user might use when he enters his question, so it gets the highest weight. The question field should be considered but with a lower weight than the keyword field. We will also include the answer field in the scoring profile since we can assume that some of the words that appear in the question will also appear in the answer, but it gets the lowest weight.<\/p>\n<p>To simplify the process of creating this solution, we will use a small QnA corpus. Since we use a small corpus, it may seem that we can get good results without using the scoring profile feature, but this is only because of the limited size of our corpus. The more items the corpus contains, the scoring profile feature will be more powerful and provide more accurate results.<\/p>\n<p><strong>The following steps outline how to build a QnA service with Azure Search:<\/strong><\/p>\n<p>Open the Azure Portal and create a new resource group. Under this resource group, create a new Azure Search service as demonstrated here:\n<img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-1.png\" alt=\"Screenshot of Azure Portal resource group. Adding new service Azure Search from services list.\" \/><\/p>\n<p>Provide a name for your search service and fill in the rest of the required fields, then press OK:\n<img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-2.png\" alt=\"Creating new search service with URL, associated subscription, resource group, location and pricing tier.\" \/><\/p>\n<p>This is how it looks like when our Azure Search service is deployed:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-3.png\" alt=\"Overview of resource group service is in showing search its name, type, and location.\" \/><\/p>\n<p>Azure Search can import data from a few different data sources such as Document DB, SQL, or Azure Table. In this example I created a SQL DB with one table and used it as the source for the search service:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-4.png\" alt=\"Create a SQL DB and server\" \/><\/p>\n<p>Fill in required fields:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-5.png\" alt=\"Create a SQL DB\" \/><\/p>\n<p>This is how the final result should look:<\/p>\n<p><img decoding=\"async\" src=\"\/developerblog\/wp-content\/uploads\/2016-12-10-Azure-Search-QNA-6.png\" alt=\"SQL DB Ready\" \/><\/p>\n<p>I used SQL Server Management Studio to connect to my SQL DB. I then created a table using the following query:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-7.png\" alt=\"Creating SQL table\" \/><\/p>\n<p>Then I filled in the table with a few questions and answers:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-8.png\" alt=\"Filled in table with Q&amp;A items\" \/><\/p>\n<p>Now that the data is ready, it\u2019s time to import it into the Search service. Follow the red squares to import the data from SQL:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-9.png\" alt=\"Import data for Azure Search\" \/><\/p>\n<p>After connecting to the SQL, as part of the data import wizard, it\u2019s time to define the index properties. Provide a name for the index, and select the ID field in the Key select box.<\/p>\n<p>Next, we\u2019ll need to select a few properties for each field in our table:<\/p>\n<ul>\n<li><code class=\"highlighter-rouge\">Retrievable<\/code> should be checked if we want to retrieve this field as part of the search result<\/li>\n<li><code class=\"highlighter-rouge\">Filterable<\/code> should be selected if we want to be able to filter based on this field<\/li>\n<li><code class=\"highlighter-rouge\">Sortable<\/code> should be checked if we want to be able to sort based on this field<\/li>\n<li><code class=\"highlighter-rouge\">Facetable<\/code> should be checked if we want to be able to get counters for the number of results grouped by this field. We\u2019ll see that in action next.<\/li>\n<li><code class=\"highlighter-rouge\">Searchable<\/code> should be used to mark the fields we would like the search engine to use when performing the actual search<\/li>\n<\/ul>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-11.png\" alt=\"Build the indexer\" \/><\/p>\n<p>The next step will be to define the analyzers that will be used for each field. In this case, we used <code class=\"highlighter-rouge\">English-Microsoft<\/code> for all of the searchable fields. Use the <strong><em>Language<\/em>-Microsoft<\/strong> value based on the language you use:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-12.png\" alt=\"Analyzers\" \/><\/p>\n<p>For this demo, I chose <code class=\"highlighter-rouge\">Once<\/code> for the scheduler, but this is the place where you can provide scheduling details for when the service should re-index your data for changes and updates:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-13.png\" alt=\"Index name\" \/><\/p>\n<p>Click OK. When finished, you should receive a notification that the import process completed successfully.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-14.png\" alt=\"Complete import process\" \/><\/p>\n<p>Next, follow the red squares in the screenshot below to create your scoring profile:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-15.png\" alt=\"Scoring profile\" \/><\/p>\n<p>Next, we will need to provide weights for the different fields, as explained above:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-16.png\" alt=\"Weights\" \/><\/p>\n<p>We are now ready to start searching our QnA corpus. Click the Search explorer button as demonstrated in the following screenshot:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-17.png\" alt=\"Complete import process\" \/><\/p>\n<p>The Search explorer provides an easy method to invoke REST API calls from within the portal. This point is where we can experiment using our search service.<\/p>\n<p>Use <code class=\"highlighter-rouge\">search=your_free_text_question<\/code> in the Query string input as demonstrated below. This query will invoke a simple search on the searchable fields without using the scoring profile feature.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-18.png\" alt=\"Simple Search Query\" \/><\/p>\n<p>Now let\u2019s use the scoring profile by adding it to the query string:<\/p>\n<p><code class=\"highlighter-rouge\">search=in which mode am i&amp;scoringProfile=qna-scoring-profile<\/code><\/p>\n<p><strong>Note<\/strong>: You can see that Azure Search returned different items and scores than before (the last item is 14 instead of 5 like it was when we searched without the scoring profile). It is a bit difficult to demonstrate with a small number of items how much the scoring profile helps. However, having a large number of items in the scoring profile makes a difference.<\/p>\n<p>Learn more about <a href=\"https:\/\/msdn.microsoft.com\/en-US\/library\/dn798927.aspx\">search parameters and how to call the search service API<\/a><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-19.png\" alt=\"Using Scoring Profile\" \/><\/p>\n<p>In this sample, we used the facet feature to get a counter for the number of search results under each category. This feature can be useful when there are few results for each category, by asking the user which category he would like to browse and displaying the number of results in each. After the user chooses the category, we can then drill down by filtering the category with an additional search:<\/p>\n<p><code class=\"highlighter-rouge\">search=in which mode am i?&amp;scoringProfile=qna-scoring-profile&amp;facet=category<\/code><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2016-12-10-Azure-Search-QNA-20.png\" alt=\"Using Facets\" \/><\/p>\n<h2 id=\"opportunities-for-reuse\">Opportunities for Reuse<\/h2>\n<p>The work outlined in this case study can be reused for providing language support in any bot QnA scenario. For example, this approach can be used to build a QnA system for Norwegian or Hebrew, two languages not natively supported by QnA Maker.<\/p>\n<p><strong>Note<\/strong>: This approach is not useful just for bot clients. The technique described in this post can be used to build a general, domain-specific search service, which can be tweaked further with scoring profiles.<\/p>\n<p>Additionally, because Azure Search provides the ability to label individually indexed items, the methodology above can be used as a scalable multiclass text classification method. This possibility is critical because many bots need to categorize intent based on messages from users. While solutions such as Luis or Azure ML exist, they can sometimes struggle when there are more than 10-15 classification categories. For example, if a doctor needs to classify a medical protocol out of hundreds of medical protocols, this approach can be used to classify text, given a large enough corpus of related protocols.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>How to provide a backend service for a QnA bot in languages not currently supported by QnA Maker by utilizing Azure Search.<\/p>\n","protected":false},"author":21349,"featured_media":1203,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[10,13],"tags":[92,110,266,301],"class_list":["post-2124","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure-app-services","category-bots","tag-azure-search","tag-bots","tag-multilingual-support","tag-qna-maker"],"acf":[],"blog_post_summary":"<p>How to provide a backend service for a QnA bot in languages not currently supported by QnA Maker by utilizing Azure Search.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/2124","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/21349"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=2124"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/2124\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=2124"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=2124"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=2124"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}