The following case study outlines a method for providing additional language support for LUIS using the Microsoft Translation Cognitive API<\/a>.<\/p>\n Moed.ai<\/a> is an Israeli Startup that enables service providers to manage and fill their business calendar with a unified cloud-based platform, that is accessible from any device.<\/p>\n Customers can configure scheduling of their services, resources and calendars using Moed.ai\u2019s designated dashboard. Resources can be objects, such as cars or meeting rooms, as well as people, such as test drivers or sales representatives in a car dealership. The platform manages a calendar for each of these resources and uses it to schedule meetings with customers\u2019 clients based on availability.<\/p>\n Moed.ai is developing a chat bot for each of their customers so that their customers\u2019 clients can schedule services more comfortably through natural language on their preferred channel (Facebook Messenger, Slack, Skype, etc.).<\/p>\n As an Israeli company, many of Moed.ai\u2019s customers are native Hebrew speakers. While they provide an English version of their bot that can extract intents and entities, they want to provide equal functionality for their Hebrew bot service. Unfortunately, LUIS<\/a>, which they were interested in using for intent and entity extraction, does not currently have native support for the Hebrew language.<\/p>\n The goal of the engagement was to work with Moed.ai to identify a valid approach for providing Hebrew support for LUIS using the Translation Cognitive Service<\/a>. During the course of the engagement, we compared two approaches for providing Hebrew support. While, the first approach of feeding translated text directly from the Translation Cognitive Service<\/a> into an existing English LUIS model provided disappointing results, we were successful in determining a more accurate method.<\/p>\n We trained the LUIS model in a novel way, using malformed translated text instead of a list of pre-curated English samples. This approach enabled us to close the gap between the translation service\u2019s output and proper English.<\/p>\n To understand why this approach works more accurately, let\u2019s look at the following scenario:<\/p>\n Assume a user wants to query a bot with the four Hebrew phrases below.<\/p>\n The English equivalent for these phrases is the following:<\/p>\n Yet the translation service translates the phrases as:<\/p>\n Note that while the first two phrases and their translations are nearly identical, there is a gap between the translation of the second two phrases (\u201cI want to make a test tomorrow.\u201d , \u201cCan set a test tomorrow?\u201d) and their proper English meaning (\u201cI want to schedule a test drive for tomorrow.\u201d, \u201cCan I schedule a test drive tomorrow?\u201d).<\/p>\n For example, in both phrases, the translation service substituted the word test<\/strong> in place of the concept test drive<\/strong>, which has a very different meaning despite being a close literal translation. A LUIS model trained only on proper English queries, such as \u201cI want to schedule a test drive for tomorrow\u201d would struggle to identify such substitutions since they are unique to the way Hebrew is translated into English. Differences in grammatical structure and word usage between different languages often lead to consistent but unique errors in translated texts.<\/p>\n However, if we train the model on translated Hebrew, the service will quickly learn to identify the gaps between the malformed Hebrew translation and its intended meaning. Over time, as the model learns the unique ways in which Hebrew translations are erroneous in a given context, it will provide more and more accurate results.<\/p>\n The following section outlines how to train and use our node module for integrating additional language support for bots. It is assumed that the user has already created a LUIS<\/a> application and has generated a key for the Translation Cognitive Service<\/a>.<\/p>\n 1) Compile a list of commands in the unsupported language (in this case Hebrew) such as:<\/p>\n 2) Run the Bulk Translate and Insert into LUIS<\/a> Script<\/p>\n 3) Tag translations, intents and entities using the LUIS portal<\/a><\/p>\n 4) Use the train and test bot<\/a> with the LUIS portal<\/a> to validate and re-train your model until it learns to fit the translations to the new language meanings.<\/p>\nBackground<\/h2>\n
The Problem<\/h2>\n
The Solution<\/h2>\n
\u05d0\u05e0\u05d9 \u05e8\u05d5\u05e6\u05d4 \u05dc\u05e7\u05d1\u05d5\u05e2 \u05e4\u05d2\u05d9\u05e9\u05d4\r\n \u05d0\u05e0\u05d9 \u05e8\u05d5\u05e6\u05d4 \u05dc\u05e7\u05d1\u05d5\u05e2 \u05e0\u05e1\u05d9\u05e2\u05ea \u05de\u05d1\u05d7\u05df \r\n \u05d0\u05e0\u05d9 \u05e8\u05d5\u05e6\u05d4 \u05dc\u05e7\u05d1\u05d5\u05e2 \u05e0\u05e1\u05d9\u05e2\u05ea \u05de\u05d1\u05d7\u05df \u05dc\u05de\u05d7\u05e8\r\n \u05d0\u05e4\u05e9\u05e8 \u05dc\u05e7\u05d1\u05d5\u05e2 \u05e0\u05e1\u05d9\u05e2\u05ea \u05de\u05d1\u05d7\u05df \u05dc\u05de\u05d7\u05e8?\r\n\r\n<\/code><\/pre>\n<\/div>\nI want to schedule a meeting.\r\nI want to schedule a test drive.\r\nI want to schedule a test drive for tommorrow.\r\nCan I schedule a test drive tomorrow?\r\n\r\n<\/code><\/pre>\n<\/div>\nI want to schedule an appointment.\r\nI want to schedule a test drive.\r\nI want to make a test tomorrow.\r\nCan set a test tomorrow?\r\n\r\n<\/code><\/pre>\n<\/div>\nHow to Use<\/h3>\n
\u05d0\u05e0\u05d9 \u05e8\u05d5\u05e6\u05d4 \u05dc\u05e7\u05d1\u05d5\u05e2 \u05e4\u05d2\u05d9\u05e9\u05d4 \/\/ I want to schedule an appointment\r\n \u05d0\u05e0\u05d9 \u05e8\u05d5\u05e6\u05d4 \u05dc\u05e7\u05d1\u05d5\u05e2 \u05e0\u05e1\u05d9\u05e2\u05ea \u05de\u05d1\u05d7\u05df \/\/ I want to schedule a test drive\r\n \u05d0\u05e0\u05d9 \u05e8\u05d5\u05e6\u05d4 \u05dc\u05e7\u05d1\u05d5\u05e2 \u05e0\u05e1\u05d9\u05e2\u05ea \u05de\u05d1\u05d7\u05df \u05dc\u05de\u05d7\u05e8 \/\/ I want to schedule a test drive for tomorrow\r\n \u05d0\u05e4\u05e9\u05e8 \u05dc\u05e7\u05d1\u05d5\u05e2 \u05e0\u05e1\u05d9\u05e2\u05ea \u05de\u05d1\u05d7\u05df \u05dc\u05de\u05d7\u05e8? \/\/ Can I schedule a test drive tomorrow?\r\n\r\n<\/code><\/pre>\n<\/div>\n