Combining OpenAI function calls with embeddings
Hello prompt engineers,
Last week’s post introduced the OpenAI chat function calling to implement a live weather response. This week, we’ll look at how to use function calling to enhance responses when using embeddings to retrieve data isn’t appropriate. The starting point will be the droidcon SF sample we’ve covered previously:
Figure 1: droidcon chat and the questions it can answer using the system prompt or embedding similarity
As you can see, the droidcon chat implementation can answer questions like “when is droidcon SF?” (using grounding in the system prompt) and “are there any AI sessions?” (using embedding similarity comparisons between the question and the session info). However, it can’t answer questions like “what’s on right now?” or “what’s up next?”.
The remainder of this post will discuss the obstacles and provide a solution to answering these “time-based” questions using the OpenAI chat API with function calling.
TIP: When testing this demo, open JetchatAI and tap the ‘users’ icon at the top-left to switch from the default OpenAI-chat to the droidcon-chat.
What time is it?
The first issue that the model has answering these types of questions is that it does not automatically know the date and time:
Figure 2: The model can’t answer “what’s on now” in part because it does not know the date or time
To solve this problem, we can include the date and time in each prompt (in the grounding function in DroidconEmbeddingsWrapper.kt). This additional text will be added to each prompt submitted in the chat (although it will not be shown in the UI). A basic (if slightly verbose) implementation could look like this:
val date = LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyy-MM-dd"))
val time = LocalDateTime.now().format(DateTimeFormatter.ofPattern("HH:mm")) // 24 hour format
messagePreamble =
"The current date is $date and the time (in 24 hour format) is $time.\n\n$messagePreamble"
Figure 3: Code to ground each request with the date and time
Aside: testing time-based functions…
While the code in Figure 3 would work, it would be hard to test. Assuming we’re developing this app weeks or months before the conference, testing the prompt “what’s on now” is never going to return the correct results! Instead, allow for a specific date and time to be used for testing – it’s much easier than updating the date and time on your test devices:
var date = Constants.TEST_DATE
var time = Constants.TEST_TIME
if (date == "") {
date = LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyy-MM-dd"))
}
if (time == "") {
time = LocalDateTime.now().format(DateTimeFormatter.ofPattern("HH:mm"))
}
messagePreamble =
"The current date is $date and the time (in 24 hour format) is $time.\n\n$messagePreamble"
Figure 4: Actual code used in the sample to use a hardcoded date & time when provided (for testing)
In the sample code the hardcoded values are in the Constants.kt file:
/** Hardcode date "2023-06-09" for droidcon session testing, make empty string to use real date */ internal const val TEST_DATE = "2023-06-09" /** Hardcode time "15:45" for droidcon session testing, make empty string to use real time */ internal const val TEST_TIME = "15:45"
Figure 5: Hardcoded Constants.kt values for testing specific dates and times
By hardcoding a specific date and time, we can change it to simulate different testing scenarios such as when the user is querying before the conference, during the conference, and even after the conference. You can then set it to “2023-06-01” to test how the model answers before the conference, “2023-06-08” for one of the days of the conference, and “2023-07-01” to see how it responds after the conference… basically test for all the dates and times you expect to have different results.
Don’t forget to remove the hardcoding before making release builds!
Although we haven’t implemented any unit testing in the sample, taking this idea further and allowing the date and time to be dynamically injected when unit tests run would be even more flexible.
The problem with embeddings
There are some queries that can’t be easily answered by doing an embedding similarity comparison, like “what are the sessions on right now?” or “what are the sessions tomorrow morning?”. This is because comparing the ‘subject’ or ‘intent’ of those strings is not going to have a close similarity to the session descriptions, and also because the concept of the ‘current time’ needs to be represented accurately as it’s constantly changing.
If we ask the model time-based questions after adding the current date/time grounding, it still can’t answer:
Figure 6: when no embeddings match, model has no data. This can lead to hallucinations – there is no such session or speaker as “Jane Smith”
What’s worse, you can see in Figure 6 that the model hallucinates a session that does not exist in the droidcon SF 2023 schedule! While we might be able to address the hallucination with updates to the system prompt, or changing the temperature parameter, it would be better if we could help the model to get the right answer to these questions.
Add a time-based chat function
In order to answer the question of what sessions are on at a given time, we first need to be able to query the droidcon sessions by time. This is a job best suited to a database, but for simplicity the data has been modelled as a collection of objects in DroidconSessionObjects which provides date and time properties for each session and allowing them to be filtered by a simple date and time comparison:
if (date == session.date) {
if (session.time in earliestTimeCheck..latestTimeCheck) {
out += session.toJson() + "\n" // return session details to model
Figure 7: Simple code to query the collection of session objects
Given that we can query the sessions by date and time, this can be exposed to the model by a chat function similar to the currentWeather function discussed last week. This new function can be found in the functions/SessionsByTimeFunction.kt and can be summarized as follows:
-
name– sessionByTime -
description– “Given a date and time or time range, return the sessions that start on that date, during that time” -
parameters–-
date– “Date that the conference sessions occur, eg 2023-06-08” -
earliestTime– “The earliest time that the conference sessions might start, eg. 09:00. Defaults to the current time” -
latestTime– “The latest time that the conference sessions might start, eg. 14:00. Defaults to one hour from current time.”
-
date is the only required parameter, the time restrictions are optional and will be considered to have default values if the model can’t infer them from the user’s query. The function is modeled with a time range so that questions like “what sessions start in the next hour” or “what sessions are on this morning” can be answered – in each case the model can figure out the range based on the grounding and its knowledge of time (eg. the morning ends at noon).
The code to support the function is shown in three figures below:
-
Figure 8 –
SessionsByTimeFunctionclass that encapsulates all the metadata and behavior (designed to make it easier to add more functions in future) - Figure 9 – adding the function to the chat completion request, so that the model can decide whether to call it
- Figure 10 – parsing the completion result and sending back to the model for the final result
Besides the boilerplate code to add the function to the request and response, most of the logic is handling the optional time parameters, choosing defaults, and turning a single time into a range (eg. if you ask for “what’s on now?”, the logic queries for sessions who’s start time was within the last hour – a slight hack that could be improved with math to check if the end time hasn’t also passed, if we stored that information in the data).
class SessionsByTimeFunction {
companion object {
fun name(): String {
return "sessionsByTime"
}
fun description(): String {
return "Given a date and time or time range, return the sessions that start on that date, during that time."
}
fun params(): Parameters {
val params = Parameters.buildJsonObject {
put("type", "object")
putJsonObject("properties") {
putJsonObject("date") {
put("type", "string")
put("description", "Date that the conference sessions occur, eg 2023-06-08")
}
putJsonObject("earliestTime") {
put("type", "string")
put("description", "The earliest time that the conference sessions might start, eg. 09:00. Defaults to the current time.")
}
putJsonObject("latestTime") {
put("type", "string")
put("description", "The latest time that the conference sessions might start, eg. 14:00. Defaults to one hour from current time.")
}
}
putJsonArray("required") {
add("date")
}
}
return params
}
/**
* Filter sessions by the date/time they start
*/
fun function(date: String, earliestTime: String="08:00", latestTime: String="18:00"): String {
Log.i("LLM", "sessionsByTime ($date, $earliestTime, $latestTime)")
var earliestTimeCheck = earliestTime
var latestTimeCheck = latestTime
// Always make the start time longer than a session, to get 'current' sessions
val earliest = LocalDateTime.parse("$date $earliestTime", DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm"));
val earliestLessOneHour = earliest.minusHours(1)
earliestTimeCheck = earliestLessOneHour.format(DateTimeFormatter.ofPattern("HH:mm"))
if (latestTimeCheck == "") { // only calc later time if blank
val earliest = LocalDateTime.parse("$date $earliestTime", DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm"));
val earliestPlusOneHour = earliest.plusHours(1)
latestTimeCheck = earliestPlusOneHour.format(DateTimeFormatter.ofPattern("HH:mm"))
}
val list = SessionInfo.hardcodedSessionsList()
var out: String = ""
for (session in DroidconSessionObjects.droidconSessions.values)
{
if (date == session.date) {
if (session.time in earliestTimeCheck..latestTimeCheck) {
out += session.toJson() + "\n"
}
}
}
if (out == "") {
// this is what’s returned before/after the conference
out = "There are no sessions available. Remind the user about the dates and times the conference is open."
}
return out
}
}
}
Figure 8: All the code relating to the function encapsulated in one class
With the function name, description, and parameters all in a single object, the code to add the function to the chat completion is easy to read:
val chatCompletionRequest = chatCompletionRequest {
model = ModelId(Constants.OPENAI_CHAT_MODEL)
messages = conversation
functions {
function {
name = SessionsByTimeFunction.name()
description = SessionsByTimeFunction.description()
parameters = SessionsByTimeFunction.params()
}
}
functionCall = FunctionMode.Auto
}
Figure 9: Add the function call to the chat completion request
The final step is handling the function call for the model. This consists mostly of detecting the function name, parsing the required and optional parameters safely, and then calling the function itself:
when (function.name) {
SessionsByTimeFunction.name() -> {
val functionArgs = function.argumentsAsJson() ?: error("arguments field is missing")
var optionalEarliestTime = ""
var optionalLatestTime = ""
if (functionArgs.containsKey("earliestTime")) {
optionalEarliestTime = functionArgs.getValue("earliestTime").jsonPrimitive.content
}
if (functionArgs.containsKey("latestTime")) {
optionalLatestTime = functionArgs.getValue("latestTime").jsonPrimitive.content
}
// DO THE FUNCTION
functionResponse = SessionsByTimeFunction.function(
functionArgs.getValue("date").jsonPrimitive.content,
optionalEarliestTime,
optionalLatestTime
)
} // ...
Figure 10: Handle the function call by parsing out the parameters (checking if the optional parameters exist)
In the function implementation shown in Figure 9, the response string functionResponse is what is added to the next chat completion request. The model uses that information to ultimately determine how it responds to the user’s initial query. The session.toJson() function is not shown but does what you’d expect and emits the session object as a simple JSON key-value collection. The model doesn’t actually require a specific data format or even data set, it’s up to the implementor to extract what’s likely to be the most useful information and include it in the prompt. However, remember that metadata is useful for the model, so things like the key names will help provide context that can improve the result of the query.
Querying for sessions by time
The model is now grounded with the current date and time, and because it has knowledge of a function with date and time parameters it correctly infers when to “call” it and what parameters to pass. The implementation further inspects the parameters (especially the time) to query the data and return a sensible set of session information.
The model parses the returned JSON and formats appropriately (notice that in the “what’s on now” answer it does not provide the time, but for the “what’s up next” answer it does 😉). With the date and time hardcode for the final afternoon of droidcon SF 2023, these queries return the following results:
Figure 11: The data returned from the function can be used to answer the questions “what’s on now?” and “what’s up next?”
Now that the function is available, each user query will be evaluated both against the session embeddings and the function (or functions) available. It’s possible that one or the other or both will return data in the chat completion, and it will always be up to the model to make a final determination of how to combine and format the answer to the user’s original query.
Resources and feedback
You can find the code for this post on the JetchatAI repo.
If you have any questions, use the feedback forum or message us on Twitter @surfaceduodev.
There will be no livestream this week, but you can check out the archives on YouTube.
Light
Dark
0 comments