\n I presented a session<\/a> at droidcon SF<\/a> last week, so it seemed appropriate to take our existing JetchatAI sample and enable it to answer questions about the conference schedule<\/a>.\n<\/p>\n \n RAG<\/a> is short for Retrieval Augmented Generation (see also this paper from Meta<\/a>) and describes a pattern for querying LLMs where we first examine the user\u2019s input and try to determine what data they\u2019re interested in, and then if needed pre-fetch related information to include as \u2018context\u2019 in the actual request to the model. \n<\/p>\n \n This additional context could come from the internet, a database, or any other query-able source, including internal documentation or personal content.\n<\/p>\n \n The hard part is figuring out, from the user\u2019s input, exactly what information do we need to pre-fetch? There could be a number of approaches, from a custom model trained on extracting query intent for a given use-case, to something simple like extracting keywords and conducting a search. Some different approaches for generating responses are discussed in this blog about using OpenAI with Azure Cognitive Search<\/a>.\n<\/p>\n NOTE: A common question on this approach is \u201conce you\u2019ve pre-fetched information, why not just show that to the user? Why send the augmented query to the LLM at all?\u201d. The answer lies in the model\u2019s ability to summarize the collected information, hopefully ignoring irrelevant data, and crafting a text response that most directly answers the original question.<\/p><\/blockquote>\n \n This diagram from the Azure blog helps to visualize the process:\n<\/p>\n \n \n For the droidcon demo I extracted the session information into a collection of text chunks to serve as the data source. To determine whether there were any relevant conference sessions in a given user question, I used another LLM feature: embeddings.\n<\/p>\n \n The OpenAI blog has a great introduction to embeddings<\/a>. An embedding is a numerical representation of content within an LLMs conceptual space, a vector with hundreds or thousands of dimensions. OpenAI has an embedding endpoint that will return the vector for any text input.\n<\/p>\n \n This visualization from the OpenAI blog shows how text snippets with related concepts cluster together in the embedding space. The number of dimensions in the vectors has been mathematically reduced from 2,048 to 3 dimensions to make it easier to read. Each colored point on the chart represents one or two sentences, such as \u201cPhil Gilbert (born 15 November 1969) is an Australian rules footballer. He played for both the Melbourne and Freemantle Football clubs in the Australian Football League\u201d or \u201cTSS Olga was a steam turbine cargo vessel operated by the London and North Western Railway from 1887 to 1908\u201d. The chart shows how the embeddings (the vector, or coordinates) for similar concepts are clustered together in the embedding space, and color coded to show how text about different categories is distributed.\n<\/p>\n \n \n This property of embeddings \u2013 where similar concepts are \u201cclose\u201d to each other \u2013 can be used evaluate similarities between two chunks of text using vector operations like dot product. For additional chunks of text, we can create an embedding vector, \u201cmap\u201d it in the visualization, and determine whether that text fits into one of these five categories based on how close it is to other embeddings.\n<\/p>\n \n To extend JetchatAI to be able to answer questions about the droidcon schedule:\n<\/p>\n \n Although you can use vector databases to efficiently store embeddings and their associated content, this simple demo just stores the data in memory using a \n There are about 70 sessions in the example file DroidconSessionData.kt<\/a><\/strong>.<\/a>\n<\/p>\n \n The code to generate embeddings is in the \n This code uses the OpenAI embedding endpoint to loop through all the sessions and create an embedding vector to be stored in memory:\n<\/p>\n \n Creating an embedding for each message is done in the \n When the similarity (calculated by the dot product) is above a certain threshold (0.8 for this demo), the session text is included in the prompt that will be sent to the LLM.\n<\/p>\n \n This code builds the augmented prompt, adding additional context and instructions at the end.\n<\/p>\n \n The initial chat message will be added to the end of this prompt before being appended to the \n The response from the LLM is displayed directly in the chat. If the prompt was augmented with additional data, the response will probably include some of that information, although the model may also decide to ignore information that it doesn\u2019t feel relevant.\n<\/p>\n \n The additional instructions added to the augmented prompt \u2013 \n You can download and try out the sample from the droidcon-sf-23 repo<\/a> by adding your own OpenAI API key<\/a>.<\/p>\n NOTE: A number of shortcuts have been created to keep this demo simple \u2013 such as calculating embeddings on-the-fly, including the session metadata for embedding, picking an arbitrary 0.8 cutoff to test similarity, and probably other hacks. While it shows how easy it is to bring additional data to LLM responses, please look for better solutions if you take the next step and start incorporating LLM chat into your production apps.<\/p><\/blockquote>\n \n The code for this sample and the others that were presented at droidcon SF 2023 is available on GitHub<\/a>.\n<\/p>\n \n If you have any questions, use the feedback forum<\/a> or message us on Twitter @surfaceduodev<\/a>.\n<\/p>\n![\"Android](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/06\/word-image-3284-1.png\")
It\u2019s RAG time<\/h2>\n
\n
![\"Revolutionize](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/06\/thumbnail-image-1-of-blog-post-titled.png\")
What are embeddings?<\/h2>\n
![](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/06\/word-image-3284-3.png\")
![](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/06\/word-image-3284-4.png\")
Use embeddings to augment chat completions<\/h2>\n
\n
1. Session \u201cdatabase\u201d<\/h3>\n
Map<\/code>. An example of the raw session data is shown below. Note that the data has been semi-structured as key-value pairs \u2013 this helps the LLM understand context when formulating its final response.\n<\/p>\n val droidconSessions: Map<String, String> = mapOf<\/em>(\r\n \"craig-dunn\" to <\/em>\"\"\"Speaker: CRAIG DUNN\r\n Role: Software Engineer at Microsoft\r\n Location: Robertson 1\r\n Date: 2023-06-09\r\n Time: 16:30\r\n Subject: AI for Android on- and off-device\r\n Description: AI and ML bring powerful new features to app developers, for processing text, images, audio, video, and more. In this session we\u2019ll compare and contrast the opportunities available with on-device models using ONNX and the ChatGPT model running in the cloud.\"\"\"\r\n \/\/...<\/pre>\n
2. Generate embeddings<\/h3>\n
initVectorCache<\/code> method in DroidconEmbeddingsWrapper.kt<\/strong><\/a>. Once again, in a production app you would pre-calculate and store these in a database of some kind. For this demo, they are calculated on-the-fly.\n<\/p>\n for (session in DroidconSessionData.droidconSessions) {\r\n val embeddingRequest = EmbeddingRequest(\r\n model = ModelId(\"text-embedding-ada-002\"),\r\n input = listOf<\/em>(session.value)\r\n )\r\n val embeddingResult = openAI.embeddings(embeddingRequest)\r\n val vector = embeddingResult.embeddings[0].embedding.toDoubleArray<\/em>()\r\n vectorCache[session.key] = vector\r\n }<\/pre>\n3. Process new chat messages<\/h3>\n
grounding<\/code> method using the same OpenAI endpoint. The message vector is then compared against the embeddings for each session \u2013 similar to an inefficient index lookup in a database. The results are sorted so that the best matches can be easily extracted:\n<\/p>\n\r\n for (session in vectorCache) {\r\n val v = messageVector dot <\/em>session.value\r\n sortedVectors[v] = session.key\r\n }\r\n<\/pre>\n4. Construct augmented prompt<\/h3>\n
if (sortedVectors.lastKey() > 0.8) { \/\/ arbitrary match threshold<\/em>\r\n messagePreamble =\r\n \"Following are some talks\/sessions scheduled for the droidcon San Francisco conference in June 2023:\\n\\n\"\r\n for (dpKey in sortedVectors.tailMap(0.8)) {\r\n messagePreamble += DroidconSessionData.droidconSessions[dpKey.value] + \"\\n\\n\"\r\n }\r\n messagePreamble += \"\\n\\nUse the above information to answer the following question. Summarize and provide date\/time and location if appropriate.\\n\\n\"\r\n }<\/pre>\nconversation<\/code> data structure and sent to the model. The augmented prompt is never shown in the app.\n<\/p>\n5. Show response<\/h3>\n
\"Summarize and provide date\/time and location if appropriate\"<\/code> \u2013 help the model give better answers by ensuring this information is included each time.\n<\/p>\n![](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/06\/word-image-3284-6.png\")
\n ![](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/06\/word-image-3284-5.png\")
\n<\/p>\nResources and feedback<\/h2>\n