\n However, to include MORE relevant context from a chat history, different approaches are required, such as history summarization or using embeddings of past context.\n<\/p>\n
\n In this post, we\u2019ll consider how summarizing the conversation history that\u2019s beyond the sliding window can help preserve context available to answer future queries while still keeping within the model\u2019s token limit.\n<\/p>\n
The test case<\/h2>\n
\n In the droidcon sample, one possible query that requires a lot of historical context could be “list all the sessions discussed in this chat”<\/strong>. Ideally, this would be able to list ALL the sessions in the conversation, not just the ones still mentioned in the sliding window. To test this, we\u2019ll need to first chat back-and-forth with the model, asking questions and getting responses until the sliding window is full.\n<\/p>\n An interesting side-effect of all the features we already added is that queries like this trigger the \n To make testing easier, I wrote a test conversation to logcat<\/strong> and then hardcoded it to simulate a longer conversation (the initial generation of the test data proved that the \u2018history accumulation\u2019 code worked). The test conversation can be seen in this commit<\/a>.\n<\/p>\n \n Older messages that exist outside of the sliding window need to be \u2018compressed\u2019 somehow so that they can still be included in API calls but under the token limit. To perform this \u2018compression\u2019, we will do an additional LLM completion to generate a summary of as many messages as we can that fit inside a separate completion request (which will be limited by its own maximum token size).\n<\/p>\n \n The message history will be plain text formatted, so the responses will be prefixed with \u201cUSER:\u201d or \u201cASSISTANT:\u201d. Here\u2019s an example of the format<\/a>:\n<\/p>\n Figure 1: Example text representation of chat history<\/em>\n<\/p>\n \n Conversation history like this, that extends beyond the sliding window, is what we need to succinctly summarize to keep \u2018context\u2019 while still using our token limit efficiently.\n<\/p>\n NOTE: the prompt used to generate the summary will have a big impact on the success of the feature. With this in mind, I tested a number of different prompts directly in the OpenAI Playground<\/a> before working on the Android sample in Kotlin.<\/p><\/blockquote>\n \n \n The following table shows some of the different prompts that were tested on the hardcoded sample conversation<\/a>:\n<\/p>\nAskDatabaseFunction<\/code> from Dynamic Sqlite queries with OpenAI chat functions<\/a>. The model generates a SELECT * FROM sessions<\/code> query and attempts to answer with ALL the sessions in the database. For simplicity, I have commented-out that function in DroidconEmbeddingsWrapper.kt<\/strong> for the purposes of testing the history summarization feature discussed in this post. Tweaking the prompt for that function will be an exercise for another time…<\/p><\/blockquote>\nSummarizing the message history<\/h2>\n
USER: are there any sessions about AI?\r\nASSISTANT: Yes, there is a session about AI titled \u201cAI for Android on- and off- device\u201d presented by Craig Dunn, a software engineer at Microsoft. The session will take place on June 9th at 16:30 in Robertson 1.\r\nUSER:...<\/pre>\n
![\"Screenshot](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/09\/a-screenshot-of-a-computer-description-automatica.png\")
![\"Screenshot](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/09\/a-screenshot-of-a-chat-description-automatically-3.png\")