Basic RAG embedding<\/h2>\n
\n When we first wrote<\/a> about building a Retrieval Augmented Generation (RAG) chat feature, we created a \u2018chunk\u2019 of information for each conference session. This text contains all the information we have about the session, and it was used to:\n<\/p>\n \n Figure 1 shows an example of how the text was formatted (with key:value pairs) and the types of information provided:\n<\/p>\n Figure 1: an example of the session description data format used in the original Jetchat sample app<\/em>\n<\/p>\n \n Using all the information fields for embeddings and<\/em> grounding worked fine for our use case, and we\u2019ve continued to build additional features like sliding window and history caching based on this similarity matching logic. However, that doesn\u2019t mean it couldn\u2019t be further improved!\n<\/p>\n \n When you consider how the embedding vector is used \u2013 to compare against the user query to match on general topics and subject similarity \u2013 it seems like we could simplify the information we use to create the embedding, such as removing<\/em> the speaker name, date, time, and location keys and values. This information is not well suited<\/a> to matching embeddings anyway (chat functions<\/a> and dynamic SQL querying<\/a> work much better for questions about those attributes), so we can reduce the text chunk used for generating embeddings to the speaker role, subject, and description as shown in Figure 2:\n<\/p>\n Figure 2: a more focused text chunk for embedding. The role was included since for this dataset and expected query usage it often contains relevant context.<\/em>\n<\/p>\n \n There is an immediate improvement in cost efficiency, as the new text chunk is only 73 tokens, versus 104 tokens for the complete text in Figure 1 (which would be a 25% saving in the cost of calculating all the embeddings; although note that some have longer descriptions than others so the amount of cost savings will vary). While embedding API calls<\/a> are much cheaper (at $0.0004 per 1000 tokens) than the chat API<\/a> ($0.002 to $0.06 per 1000 tokens), it\u2019s still a cost that can add up over time so it makes sense to reduce the amount of tokens used to create embeddings if possible.\n<\/p>\n Note that the shorter text chunk is ONLY used for creating the embedding vector. When the vector similarity with the user query is high enough, the ORIGINAL text with all the fields is what is added to the chat prompt grounding. This ensures that the chat model can still respond with speaker, date, time, and location information in the chat.<\/strong>\n<\/p>\n \n Testing with some common user queries from other blog posts in this series, the vector similarity scores are very close when comparing the query embedding vector against the larger text chunk (old score) and the smaller text chunk (new score). About a quarter of the sample were slightly lower scoring (highlighted in red), but the rest resulted in higher similarity scores. \n<\/p>\n User query<\/strong>\n<\/p>\n<\/td>\n Matching session<\/strong>\n<\/p>\n<\/td>\n Old similarity score<\/strong>\n<\/p>\n<\/td>\n New similarity score<\/strong>\n<\/p>\n<\/td>\n<\/tr>\n \n Are there any sessions on AI\n<\/p>\n<\/td>\n \n AI for Android on- and off-device\n<\/p>\n<\/td>\n \n 0.807\n<\/p>\n<\/td>\n \n 0.810\n<\/p>\n<\/td>\n<\/tr>\n \n Are there any sessions on gradle\n<\/p>\n<\/td>\n \n Improving Developer Experience with Gradle Build Scans (Rooz Mohazzabi)\n<\/p>\n<\/td>\n \n 0.802\n<\/p>\n<\/td>\n \n 0.801\n<\/p>\n<\/td>\n<\/tr>\n \n Improving Developer Experience with Gradle Build Scans\n<\/p>\n<\/td>\n \n 0.810\n<\/p>\n<\/td>\n \n 0.806\n<\/p>\n<\/td>\n<\/tr>\n \n Improving Developer Experience with Gradle Build Scans (Iury Souza)\n<\/p>\n<\/td>\n \n 0.816\n<\/p>\n<\/td>\n \n 0.823\n<\/p>\n<\/td>\n<\/tr>\n \n Crash Course in building your First Gradle Plugin\n<\/p>\n<\/td>\n \n 0.821\n<\/p>\n<\/td>\n \n 0.827\n<\/p>\n<\/td>\n<\/tr>\n \n Are there any sessions on Jetpack Compose\n<\/p>\n<\/td>\n \n Material You Review\n<\/p>\n<\/td>\n \n 0.802\n<\/p>\n<\/td>\n \n 0.801\n<\/p>\n<\/td>\n<\/tr>\n \n Building a component library in Compose for a large-scale banking application\n<\/p>\n<\/td>\n \n 0.814\n<\/p>\n<\/td>\n \n 0.824\n<\/p>\n<\/td>\n<\/tr>\n \n Developing Apps optimized for Wear OS with Jetpack Compose\n<\/p>\n<\/td>\n \n 0.815\n<\/p>\n<\/td>\n \n 0.814\n<\/p>\n<\/td>\n<\/tr>\n \n Animating content changes with Jetpack Compose\n<\/p>\n<\/td>\n \n 0.819\n<\/p>\n<\/td>\n \n 0.827\n<\/p>\n<\/td>\n<\/tr>\n \n Practical Compose Navigation with a Red Siren\n<\/p>\n<\/td>\n \n 0.823\n<\/p>\n<\/td>\n \n 0.825\n<\/p>\n<\/td>\n<\/tr>\n \n Compose-View Interop in Practice\n<\/p>\n<\/td>\n \n 0.824\n<\/p>\n<\/td>\n \n 0.838\n<\/p>\n<\/td>\n<\/tr>\n \n Panel Discussion: Adopting Jetpack Compose @ Scale (Christina Lee)\n<\/p>\n<\/td>\n \n 0.829\n<\/p>\n<\/td>\n \n 0.842\n<\/p>\n<\/td>\n<\/tr>\n \n Panel Discussion: Adopting Jetpack Compose @ Scale (Alejandro Sanchez)\n<\/p>\n<\/td>\n \n 0.831\n<\/p>\n<\/td>\n \n 0.849\n<\/p>\n<\/td>\n<\/tr>\n \n Creative Coding with Compose: The Next Chapter\n<\/p>\n<\/td>\n \n 0.832\n<\/p>\n<\/td>\n \n 0.840\n<\/p>\n<\/td>\n<\/tr>\n \n Panel Discussion: Adopting Jetpack Compose @ Scale (Vinay Gaba)\n<\/p>\n<\/td>\n \n 0.834\n<\/p>\n<\/td>\n \n 0.850\n<\/p>\n<\/td>\n<\/tr>\n<\/table>\n Figure 4: comparing vector similarity scores with the full text chunk embedding versus (old) the shorter version (new). Scores truncated to three decimal places for clarity.<\/em>\n<\/p>\n \n The results seem to show the arbitrary cut-off of \u201c0.8\u201d for measuring whether a session was a good match seemed to still apply, and the actual results displayed to the user was unchanged.\n<\/p>\n \n Since (for these test cases at least) the chat responses in the app are identical, this improvement hasn\u2019t affected the user experience positively or negatively (but it has reduced our embeddings API costs). Further testing on other conference queries might reveal different effects, and certainly for other use cases (such as reasoning over long documents using embedding chunks), \u201csummarizing\u201d the text used for embedding to better capture context that will match expected user queries could lead to better chat completions.\n<\/p>\n \n I\u2019ve left the code changes to the end of the post, since very few lines of code were changed! You can see in the pull request<\/a> that the key updates were:\n<\/p>\n These changes are very specific to our conference session data source. When the source data is less structured, you might consider generating an LLM completion to summarize each text chuck before generating the embedding.<\/p>\n \n We\u2019d love your feedback on this post, including any tips or tricks you\u2019ve learned from playing around with ChatGPT prompts.\n<\/p>\n \n If you have any thoughts or questions, use the feedback forum<\/a> or message us on Twitter @surfaceduodev<\/a>.\n<\/p>\n\n
Speaker: Craig Dunn\r\nRole: Software Engineer at Microsoft\r\nLocation: Robertson 1\r\nDate: 2023-06-09\r\nTime: 16:30\r\nSubject: AI for Android on- and off-device\r\nDescription: AI and ML bring powerful new features to app developers, for processing text, images, audio, video, and more. In this session we\u2019ll compare and contrast the opportunities available with on-device models using ONNX and the ChatGPT model running in the cloud.<\/pre>\n
More efficient embeddings<\/h2>\n
Speaker role: Software Engineer at Microsoft\r\nSubject: AI for Android on- and off-device\r\nDescription: AI and ML bring powerful new features to app developers, for processing text, images, audio, video, and more. In this session we\u2019ll compare and contrast the opportunities available with on-device models using ONNX and the ChatGPT model running in the cloud.<\/pre>\n
![\"Screenshot](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/07\/a-screenshot-of-a-chat-description-automatically-10.png\")
<\/a>Better results?<\/h2>\n
\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n Code<\/h2>\n
\n
SessionInfo<\/code> class, the first one to emit the shorter text chunk for embedding, and the larger one to emit the full text for grounding:<\/p>\nfun forEmbedding () : String {\r\n return \"Speaker role: $role\\nSubject: $subject\\nDescription:$description\"\r\n}\r\nfun toRagString () : String {\r\n return \"\"\"\r\n Speaker: $speaker\r\n Role: $role\r\n Location: $location\r\n Date: $date\r\n Time: $time\r\n Subject: $subject\r\n Description: $description\"\"\".trimIndent()\r\n}\r\n<\/pre>\n<\/li>\nDroidconEmbeddingsWrapper.initVectorCache<\/code> uses the DroidconSessionObjects<\/code> collection and the forEmbedding<\/code> function to create embeddings vectors with the summarized session info:<\/p>\nfor (session in DroidconSessionObjects.droidconSessions) {\r\n val embeddingRequest = EmbeddingRequest(\r\n model = ModelId(Constants.OPENAI_EMBED_MODEL),\r\n input = listOf(session.value.forEmbedding())\r\n )<\/pre>\n<\/li>\nDroidconEmbeddingsWrapper.grounding()<\/code> function uses toRagString()<\/code> to use the full text in the chat prompt:<\/p>\nmessagePreamble += DroidconSessionObjects.droidconSessions[dpKey.value]?.toRagString()<\/pre>\n<\/li>\n<\/ol>\n
Resources and feedback<\/h2>\n