Surface Duo Blog

Infinite chat with history embeddings

September 21, 2023 Sep 21, 2023 09/21/23
Craig Dunn
Hello prompt engineers, The last few posts have been about the different ways to create an ‘infinite chat’, where the conversation between the user and an LLM model is not limited by the token size limit and as much historical context as possible can be used to answer future queries. We previously covered: Sliding window...

“Infinite” chat with history summarization

September 14, 2023 Sep 14, 2023 09/14/23
Craig Dunn
Hello prompt engineers, A few weeks ago we talked about token limits on LLM chat APIs and how this prevents an infinite amount of history being remembered as context. A sliding window can limit the overall context size, and making the sliding window more efficient can help maximize the amount of context sent with each new chat query...

De-duplicating context in the chat sliding window

September 7, 2023 Sep 7, 2023 09/7/23
Craig Dunn
Hello prompt engineers, Last week’s post discussed the concept of a sliding window to keep recent context while preventing LLM chat prompts from exceeding the model’s token limit. The approach involved adding context to the prompt until we've reached the maximum number of tokens the model can accept, then ignoring any remaining ...

Infinite chat using a sliding window

August 31, 2023 Aug 31, 2023 08/31/23
Craig Dunn
Hello prompt engineers, There are a number of different strategies to support an ‘infinite chat’ using an LLM, required because large language models do not store ‘state’ across API requests and there is a limit to how large a single request can be. In this OpenAI community question on token limit differences in API ...

OpenAI tokens and limits

August 24, 2023 Aug 24, 2023 08/24/23
Craig Dunn
Hello prompt engineers, The Jetchat demo that we’ve been covering in this blog series uses the OpenAI Chat API, and in each blog post where we add new features, it supports conversations with a reasonable number of replies. However, just like any LLM request API, there are limits to the number of tokens that can be processed, and ...

Prompt engineering tips

August 17, 2023 Aug 17, 2023 08/17/23
Craig Dunn
Hello prompt engineers, We’ve been sharing a lot of OpenAI content the last few months, and because each blog post typically focuses on a specific feature or API, there’s often smaller learnings or discoveries that don’t get mentioned or highlighted. In this blog we’re sharing a few little tweaks that we discovered ...

Dynamic Sqlite queries with OpenAI chat functions

August 10, 2023 Aug 10, 2023 08/10/23
Craig Dunn
Hello prompt engineers, Previous blogs explained how to add droidcon session favorites to a database and also cache the embedding vectors in a database – but what if we stored everything in a database and then let the model query it directly? The OpenAI Cookbook examples repo includes a section on how to call functions with...

Embedding vector caching (redux)

August 3, 2023 Aug 3, 2023 08/3/23
Craig Dunn
Hello prompt engineers, Earlier this year I tried to create a hardcoded cache of embedding vectors, only to be thwarted by the limitations of Kotlin (the combined size of the arrays of numbers exceeded Kotlin’s maximum function size). Now that we’ve added Sqlite to the solution to support memory and querying, we can use that ...

Chat memory with OpenAI functions

July 27, 2023 Jul 27, 2023 07/27/23
Craig Dunn
Hello prompt engineers, We first introduced OpenAI chat functions with a weather service and then a time-based conference sessions query. Both of those examples work well for ‘point in time’ queries or questions about a static set of data (e.g., the conference schedule). But each time the JetchatAI app is opened, it has ...

Combining OpenAI function calls with embeddings

July 20, 2023 Jul 20, 2023 07/20/23
Craig Dunn
Hello prompt engineers, Last week’s post introduced the OpenAI chat function calling to implement a live weather response. This week, we’ll look at how to use function calling to enhance responses when using embeddings to retrieve data isn’t appropriate. The starting point will be the droidcon SF sample we’ve covered ...