- \n
- \n Sliding context window\n <\/li>\n
- \n Summarization\n <\/li>\n
- \n Embeddings\n <\/li>\n<\/ul>\n
\n The thread also suggests tools like Langchain<\/a> can help to implement these approaches, but for learning purposes, we\u2019ll examine them from first principles within the context of the Jetchat sample. This week\u2019s blog discusses the first method \u2013 a sliding context window – in our Android OpenAI chat sample.\n<\/p>\n
Before we begin\u2026<\/h2>\n
\n We\u2019re going to add a sliding window feature to the droidcon conference chat code in the Jetchat sample on GitHub<\/a>.\n<\/p>\n
Configuration changes<\/h3>\n
\n The sliding window is required because without it, eventually a long conversation will hit the token limit. In order to make this easier to test, set the
OPENAI_CHAT_MODEL<\/code> in Constants.kt<\/strong> togpt-3.5-turbo<\/code> which (currently) has a 4,096 token limit. Using the model with the lowest limit will make it easier to hit the boundary condition because we\u2019ll need fewer interactions to test hitting the limit. \n<\/p>\nCalculating tokens<\/h3>\n
\n Last week\u2019s post discussed OpenAI tokens and limits<\/a>, including a link to the interactive tokenizer<\/a>. The page includes links to Python<\/a> and Javascript<\/a> code for tokenizing inputs. Unfortunately, there doesn\u2019t seem to be anything \u2018off the shelf\u2019 for Kotlin right now, so for now we\u2019re going to use a rough approximation, which you can find in the
Tokenizer<\/code> class in the Jetchat example on GitHub<\/a>.\n<\/p>\nTest data<\/h3>\n
\n The other thing that will help with testing is user inputs that have a \u2018roughly known\u2019 token size, so that we can reproduce the failure case easily and repeatably. From the examples last week<\/a>, queries for sessions about \u201cgradle\u201d and \u201cjetpack compose\u201d result in a large number of matching embeddings that use up one- to two-thousand tokens. With just those two queries, it\u2019s possible to trigger the error state:\n<\/p>\n
\n
![\"Jetchat](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/08\/a-screenshot-of-a-message-description-automatical.png\")
\n<\/p>\nFigure 1: error when token limit is exceeded<\/em>\n<\/p>\n
\n By choosing a model with a low token limit and by testing with queries that require a large number of tokens to process, we can easily reproduce the issue, then build and test a fix.\n<\/p>\n
Inexact science<\/h3>\n
\n As mentioned above, the Kotlin tokenizer is making a very rough guess based on averages, rather than parsing according to the well-known source. Similarly, because the Kotlin OpenAI client library abstracts away the serialization of arguments to and from JSON, we also don\u2019t really have access to the exact<\/em> payload that is being sent and received from the API. We also can\u2019t really predict with accuracy how many tokens to expect in our response.\n<\/p>\n
\n To get around this issue, the token calculations err on the size of overestimating, such that we\u2019re less likely to hit edge cases where the API might be send a message that exceeds the limit. With more research it would be possible to better understand the underlying message sizes and write a better tokenizer\u2026 these are left as an exercise for the reader.\n<\/p>\n
Building a sliding window (by discarding older results)<\/h2>\n
\n \u201cSliding window\u201d is just a fancy name for a FIFO (first in first out) queue. For an AI chat feature, as the number of tokens in the conversation history approach the maximum allowed by the model, we will discard the OLDEST interactions (both user messages and model responses). Those older interactions will be \u2018forgotten\u2019 by the model when it constructs responses to new queries (although there\u2019s nothing to stop the UI from continuing to show the older messages as a long conversation thread).\n<\/p>\n
Implementation in Jetchat<\/h3>\n
\n There are three new classes in the Jetchat demo project<\/a> to support the sliding window:\n<\/p>\n
- \n
- \n
Tokenizer<\/code> \u2013 a basic averages-based token counter (number of characters divided by four).\n <\/li>\n- \n
CustomChatMessage<\/code> \u2013 a wrapper for theChatMessage<\/code> class that\u2019s used to calculate the token usage for the message. As noted above, this also can only approximate the token size of the underlying JSON payload.\n <\/li>\n- \n
SlidingWindow<\/code> \u2013 helper class that contains the algorithm to loop through the conversation history, checking the token-size of each message and building a new message collection that is \u201cguaranteed\u201d to be smaller than the model\u2019s token limit.\n <\/li>\n<\/ul>\n\n The
SlidingWindow<\/code> class is used in theDroidconEmbeddingsWrapper<\/code> class right before thechatCompletionRequest<\/code> is sent, to trim down the data used as the completion request parameter. TheSlidingWindow.chatHistoryToWindow()<\/code> function performs the following steps:\n<\/p>\n- \n
- \n Set up variables and values for the algorithm.\n<\/li>\n
- \n Checks for a system message and saves it for later (otherwise it\u2019s the \u2018oldest\u2019 message and could be ignored).\n<\/li>\n
- \n Loops through the entire message history from newest to oldest, because we want to preserve the newest messages.\n<\/li>\n
- \n Adds the system message back, reorders, and returns the message \u201cwindow\u201d\n<\/li>\n<\/ol>\n
\n The code is shown here:\n<\/p>\n
fun chatHistoryToWindow (conversation: MutableList<ChatMessage>): MutableList<ChatMessage> {\r\n val tokenLimit = Constants.OPENAI_MAX_TOKENS\r\n val expectedResponseSizeTokens = 500 \/\/ hardcoded estimate of max response size we expect (in tokens)\r\n var tokensUsed = 0\r\n var systemMessage: ChatMessage? = null\r\n val tokenMax = tokenLimit - expectedResponseSizeTokens\r\n var messagesInWindow = mutableListOf<ChatMessage>()\r\n \/\/ check for system message (to preserve it even if others are removed)\r\n if (conversation[0].role == ChatRole.System) {\r\n systemMessage = conversation[0]\r\n var systemMessageTokenCount = Tokenizer.countTokensIn(systemMessage.content)\r\n tokensUsed += systemMessageTokenCount\r\n }\r\n \/\/ loop through messages until one takes us over the token limit\r\n for (message in conversation.reversed()) {\r\n if (message.role != ChatRole.System) {\r\n var m = CustomChatMessage(message.role, \"\", message.content, message.name, message.functionCall)\r\n if ((tokensUsed + m.getTokenCount()) < tokenMax) {\r\n messagesInWindow.add(message)\r\n tokensUsed += m.getTokenCount()\r\n } else {\r\n break \/\/ could optionally keep adding subsequent, smaller messages to context up until token limit\r\n }\r\n }\r\n }\r\n \/\/ add system message back if it existed\r\n if (systemMessage != null) {\r\n messagesInWindow.add(systemMessage)\r\n }\r\n \/\/ re-order so that system message is [0]\r\n var orderedMessageWindow = messagesInWindow.reversed().toMutableList()\r\n return orderedMessageWindow\r\n }<\/pre>\nFigure 2: sliding window algorithm<\/em>\n<\/p>\n
\n The usage in the
DroidconEmbeddingsWrapper.chat()<\/code> function is right before<\/em> the creation of thechatCompletionRequest<\/code>. The updated code passes the fullconversation<\/code> list of messages to thechatHistoryToWindow<\/code> function, and then uses the subset of messages (the \u201csliding window\u201d) that\u2019s returned as the parameter for thechatCompletionRequest<\/code>:\n<\/p>\n\/\/ implement sliding window\r\n val chatWindowMessages = SlidingWindow.chatHistoryToWindow(conversation)\r\n \/\/ build the OpenAI network request\r\n val chatCompletionRequest = chatCompletionRequest {\r\n model = ModelId(Constants.OPENAI_CHAT_MODEL)\r\n messages = chatWindowMessages \/\/ previously sent the entire conversation\r\n \/\/...<\/pre>\nFigure 3: snippet of code where the sliding window function is added into the existing chat feature<\/em>\n<\/p>\n
\n The remainder of the code is unchanged \u2013 subsequent model responses are added back to the main
conversation<\/code> list and the sliding window is re-calculated again for the next user query.\n<\/p>\nFixed it!<\/h3>\n
\n To \u201cprove\u201d that the new algorithm works, the conversation in Figure 4 mimics the input used to trigger the error in Figure 1. The first two user queries \u2013 about \u201cgradle\u201d and \u201cjetpack compose\u201d \u2013 each result in a large number of embedding matches being included in the prompt (exemplified by the number of results to each \u2013 five sessions listed after the first query and ten sessions listed for the second). Prior to implementing the sliding window, the third query would have been expected to trigger an error that the maximum number of tokens was exceeded, but instead the third query (about \u201cAI\u201d) succeeds:\n<\/p>\n
\n
![\"Three](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/08\/a-screenshot-of-a-phone-description-automatically.png\")
\n<\/p>\nFigure 4: a three-interaction conversation where the first two questions have large embedding matches
<\/em>\n The \u201cproof\u201d that the algorithm is working can be see in the logcat<\/strong> output (filtered byLLM-SW<\/code>) shown in figure 5. The log shows each message from newest to oldest being compared against the \u201cavailable tokens\u201d, until one of the messages has so many tokens that including it would exceed the limit (114 tokens \u2018available\u2019 under the limit, and the \u201cGradle\u201d query with embeddings requires 2033 tokens). That message is not included in the window, so the algorithm stops adding messages and returns the subset to be sent to the model.\n<\/p>\n\n
![\"Screenshot](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/08\/a-screenshot-of-a-computer-program-description-au.png\")
\n<\/p>\nFigure 5: Logcat shows the sliding window calculation (in reverse chronological order) where the initial Gradle question and embeddings are removed from the context sent to the model<\/em>\n<\/p>\n
Sliding window for functions<\/h2>\n
\n The above examples and code snippets show queries that use embeddings to augment the model\u2019s response. However, as discussed in the post on combining OpenAI functions with embeddings<\/a> and OpenAI tokens and limits<\/a>, the addition of function calls also has an impact on token usage. The local result of function calling requires another chat completion request be sent to the model to get the model\u2019s final response.\n<\/p>\n
\n To prevent the subsequent function call completion request from exceeding the token limit, we re-apply the sliding window algorithm to the updated
conversation<\/code> list, after the function-calling-messages have been added, as shown in figure 6:\n<\/p>\n\/\/ sliding window - with the function call messages we might need to remove more from the conversation history\r\n val functionChatWindowMessages = SlidingWindow.chatHistoryToWindow(conversation)\r\n \/\/ send the function request\/response back to the model\r\n val functionCompletionRequest = chatCompletionRequest {\r\n model = ModelId(Constants.OPENAI_CHAT_MODEL)\r\n messages = functionChatWindowMessages\r\n }<\/pre>\nFigure 6: re-apply the sliding window algorithm after including the function call request and local response<\/em>\n<\/p>\n
\n Recalculating the sliding window at this point may exclude even more older context from the start of the conversation to make room for the local function response to be added to the chat completion request.\n<\/p>\n
Resources and feedback<\/h2>\n
\n Upcoming posts will continue the discussion about different ways to implement a long-term chat beyond a simple sliding window.\n<\/p>\n
\n The OpenAI developer community<\/a> forum has lots of discussion about API usage and other developer questions.\n<\/p>\n
\n We\u2019d love your feedback on this post, including any tips or tricks you\u2019ve learning from playing around with ChatGPT prompts.\n<\/p>\n
\n If you have any thoughts or questions, use the feedback forum<\/a> or message us on Twitter @surfaceduodev<\/a>.\n<\/p>\n
- \n
- \n