Since the very earlier days of Semantic Kernel, we have shipped experimental “planners” that use prompts to generate multi-step plans. This was extremely powerful because it allowed developers to use LLMs (which were created to merely generate text) to begin automating business processes.

Since then, the Semantic Kernel team has evolved its experimental planners over time so it could adopt the latest research from both inside and outside of Microsoft. Most notably, we began leveraging function calling. With function calling, the action planner could be replaced with a single function call request and the ReAct based planning of the stepwise planner could be replicated with multiple function calling steps.

As function calling has gotten increasingly more accurate and efficient, however, the need for additional “planning” logic on top of the model has become less necessary, and in some cases, can reduce the speed, cost, and accuracy of a plan.

In this blog post we’ll cover our current experimental planners and what comes next for the function calling stepwise planner and the Handlebars planner. We’ll answer: How do they work? What are they good at? And how do we plan to make them even better.

Keep an eye out for additional blog posts that provides a deep dive on how to use the future versions of the function calling stepwise planner and the Handlebars planner.

Function calling stepwise planner

As we evolved the original stepwise planner, we took advantage of function calling from OpenAI. This allowed us to reduce the size of the original prompt while still increasing accuracy. When function calling was first introduced, however, it wasn’t perfect. The LLMs had difficulty stringing together multiple function calls to complete a multi-step process (or plan) using the ReAct methodology.

To address this shortcoming, we introduced a prompt at the very beginning asking the AI to enumerate the list of steps it needed to complete a customer’s goal. This prompt was relatively lightweight and provided enough structure for function calling to work on more complex tasks.

The prompt in the function calling stepwise planner

The “better” way to ReAct

As function calling has gotten better, however, this additional step has become less necessary, and instead, has gotten in the way of functionality needed in enterprise applications…

1. Streaming was difficult to implement
2. The model had a harder time making parallel function calls
3. Reusing the entire context from a chat history object became difficult
4. And customization was difficult

So we thought to ourselves… what if we got out of the way entirely and just encouraged customers to use “vanilla” function calling. Over the last few months, the results have been astounding. Customers could achieve everything the function calling stepwise planner could with fewer tokens, more control, and significantly lower time-to-first token.

Streaming with function calling

Because of this, we’ll be sunsetting the function calling stepwise planner in favor of “vanilla” function calling. Our docs have already been updated to reflect this new recommendation, and shortly, we’ll be publishing a blog detailing how to migrate your code if you already use the function calling stepwise planner.

Handlebars planner

The Handlebars planner was the natural successor to the stepwise planner. Both were powerful because it allowed the LLM to generate an entire plan in a single LLM call. This had several benefits: a user could approve an entire “plan” before execution began and it theoretically use fewer tokens.

The challenge, however, is how do you tell the LLM how to generate a plan using as few tokens as possible? With the original sequential planner, we had to “teach” the LLM how to generate custom XML within a single prompt. This was relatively expensive and yielded poor results because the LLM couldn’t use the knowledge to generate this novel XML structure.

The prompt for the original sequential planner

The aha moment: code-based planners are more accurate

But what if we could use a language that the LLM did know how to natively write? Both TaskWeaver and AutoGen both had success using this approach, so the Semantic Kernel team had LLMs generate plans in several different programming languages (C#, Python, JavaScript) to see if it had more success. In our tests, any language performed remarkably better. The new challenge? How do you “safely” run code generated by an LLM?

Without extremely secure and limited runtime containers, we determined that this was not possible to run LLM generated code in an enterprise deployment, so we searched for a language that was purposefully very limited so the LLM couldn’t do anything it shouldn’t. We landed on the templating language Handlebars because it could do nothing more than invoke helpers. It also had the benefit of having implementations in almost all languages (meaning we could drive parity across all our SDKs)

The choice of language matters

As more customers used the Handlebars planner, we began to realize its limitations. Because the LLMs had less training data on Handlebars templates, we had to make our prompts increasingly more detailed. What originally started off as a cheaper way to generate a plan became just as token intensive as the original sequential planner.

It was around this time that Azure Container Apps released dynamic sessions. A feature that allows you to generate locked down Python containers for the explicit use of running LLM generated code. The same technology powers the Code interpreter in the Azure Assistants API. We finally had an enterprise way to run a proper coding language that the LLMs were adept at writing!

Asking LLMs to generate Python instead

LLMs are particularly good at writing Python since it’s the language of AI researchers. From the very early days of machine learning, models were trained on Python code, and because OpenAI has chosen Python as their language for Code interpreter, the model’s ability to write Python code will only get exponentially better.

Code interpreter in ChatGPT

Because of this, we will be replacing our Handlebars planner with a Python version later this fall that behaves more like OpenAI’s current Code interpreter, only with our Code interpreter, we will also allow the LLM to invoke local plugins and functions. While we develop this new planner, we recommend that new customers not use the existing Handlebars planner because it will be obsoleted.

It’s important to note that only the Handlebars planner will be deprecated. The Handlebars templating engine will remain for prompt templates.

Will a Python-based planner work with C# and Java?

The greatest concern we’ve heard with moving to a Python-based planner is whether or not this will work with the C# and Java SDK for Semantic Kernel, and for that, we have good news! Just like how we used Handlebars template generation in all three SDKs, we’ll be able to use Python code generation in all three SDKs as well.

To support local development, we’ll provide an out-of-the-box container for local testing. For production deployments, we’ll provide a connection to Azure Container Apps dynamic sessions.

We also get concern from C# and Java developers who think this will require them to author Python code. Just like the Handlebars planner, we do not expect developers to be author these plans. This is a language that only the LLM needs to know to create plans for the user during runtime and Python appears to be the best language LLMs can generate today.

Keep an eye out for an announcement in the next few months for when we release our new Python-based planning solution.

Give us feedback on our transition

If you are a current user of the existing planners and have unique scenarios you want to make sure we support with the updated planners, please reach out to us on our Semantic Kernel GitHub Discussion Channel! We want to ensure that all scenarios of the previous planners continue to be supported.