In this blog post, we show you how you can use Semantic Kernel with AutoGen, a Microsoft Research project that shows the potential of using multiple agents together. With AutoGen, Microsoft research has shown that multiple agents can be better than one.
To actually do work, however, agents within AutoGen need hooks into the real world so they can retrieve information, create content, and complete tasks. Without these capabilities, agents will simply talk to each other and waste thousands or tens-of-thousands of tokens.
This is where Semantic Kernel comes in. With Semantic Kernel, you can easily bring Plugins to agents so they can do more than just talk. They can search the web, access company resources, and ultimately become productive parts of your company.
Plugins are important in the world of “agents.” Without them, it’s impossible to get agents to complete real work.
Why would you want to work with agents? Used improperly, they can quickly consume thousands of tokens, but given the right tools (i.e., plugins) and a constrained enough task, multiple agents powered by gpt-3.5-turbo can rival the capabilities of gpt-4, all while using cheaper tokens with lower latency. While it’s still early days for multi-agent systems, here at Microsoft we believe systems like AutoGen will substantially push the AI frontier forward.
How Agents Work
Agents are based on the principle that diverse teams produce better outcomes. Greater diversity in opinion is facilitated by essentially making LLMs chat with each other. In doing so, they go beyond a “Chain of Thought”-style monologue and instead involve seeding conversations between LLMs that have different Personas.
“In addition to diversity, there’s also specialization and modularity. Agents allow us to define AI copilots as specialized (i.e. “skilled agents”) on given goals, e.g. working on a specific area, looking after security, moderating content, etc. Better yet, agents can be combined. Modularity is important because it allows agent developers to focus on one area, rather than trying to create one do-it-all AI.” —Devis Lucato, SK Founding Architect
For example, imagine one agent with a copywriter system prompt:
const string SystemMessageCopyWriter = @"
You are a copywriter with ten years of experience and are known for a brevity and a dry humor.
You are laser-focused on the goal at hand. Don't waste time with chit chat.
The goal is to refine and decide on the best copy as an expert in the field.";
Kernel kernel = Kernel.CreateBuilder().AddAzureOpenAIChatCompletion(
MyModelId, MyEndpoint, MyApiKey)
.Build();
var chatCompletion = kernel.GetRequiredService<IChatCompletionService>();
var chatHistory = new ChatHistory(SystemMessageCopyWriter);
// Add the user query to the chat history
chatHistory.AddUserMessage(userMessage);
// Finally, get the response from AI
var answer = await chatCompletion.GetChatMessageContentAsync(chatHistory);Meanwhile, another agent can be given a different persona: an art director.
var systemMessageArtDirector = "You are an art director who has opinions about copywriting
born of a love for David Ogilvy. You're laser focused on the goal at hand.
Don't waste time with chit chat. The goal is to refine and decide on the best
copy as an expert in the field."Using either Autogen or Semantic Kernel, you can then get these two agents to converse with each other. For example, the following conversation was generated by the two agents after giving them the initial marketing prompt: “maps made out of egg cartons.”
Copywriter: “Crack open a world of sustainability with our egg-carton maps. Navigate the globe, one recycled egg carton at a time.”
Art Director: The concept is intriguing, but the copy could be more direct and impactful. Let’s try: “Navigate the globe sustainably with our recycled egg-carton maps. One carton, one step towards a greener world.” This version emphasizes the product’s unique selling point and its environmental benefits more clearly.
Copywriter: “Recycled Egg-Carton Maps: Chart a greener course for our planet, one carton at a time.”
Art Director: This is a strong revision. It’s concise, direct, and clearly communicates the product’s unique selling point and environmental benefits. The phrase “Chart a greener course for our planet” is particularly effective, as it evokes a sense of adventure and responsibility. Well done.
Copywriter: Thank you. I’m glad you found the revised copy effective. If there’s anything else you need assistance with, feel free to ask.
Art Director: You’re welcome. Your revised copy is indeed effective. Keep up the good work. If you need further assistance, don’t hesitate to ask.
Using Dall-E2, the Art Director goes off and sketches up an illustration to accompany the copy:
Recycled Egg-Carton Maps: Chart a greener course for our planet, one carton at a time.
This is definitely destined for a Cannes Lion advertising industry award. Right?
More productive AutoGen agents With Semantic Kernel
AutoGen is a framework that enables the development of LLM applications using multiple agents that can converse with each other to solve tasks. AutoGen agents are customizable, conversable, and seamlessly allow human participation. They can operate in various modes that employ combinations of LLMs, human inputs, and tools.
Developing and finding the right tools to give AutoGen agents can be challenging though. Since this is an area where Semantic Kernel excels at, we wanted to demonstrate how you could give tools (i.e., plugins) to AutoGen that have already been imported into Semantic Kernel. These plugins could be authored as a collection of native functions or you could import an existing OpenAI plugin.
In our sample, we started by first creating two Semantic Kernel Plugins:
- A Bing Search Plugin, used to retrieve from Bing URLs for a given topic.
- A WebPages Plugin, used to fetch a given URL and return the text found.
We then show how you can use these two Semantic Kernel plugins within the world of AutoGen. By giving an AutoGen agent access to these plugins, the AutoGen agent can pull real-time information to answer user questions. And, because they’re the only plugins we gave it, we can ensure its working within the safe boundaries provided by the kernel.
Afterwards, a user can interact with AutoGen and it will use the plugins to:
- Search for an answer using Bing,
- Open a web page to get further details,
- Summarize the results as an answer back to the user.
Please be sure to ⭐️ (star) the GitHub repo!
Interested in learning more about agents?
There’s a vintage piece written back in 2016 for Microsoft Developer Network Magazine that describes a multi-agent approach to using AI and simulating a basketball game. If you’re looking for something more up to date, check out Lilian Weng’s piece on autonomous agents to get a sense of where this field is heading today.
Interesting read, John! I have been following the Semantic Kernel project since the beginning. We’ve even started using it internally and it’s really a great project. However, could you please clarify when to use Semantic Kernel as opposed to AutoGen? The difference isn’t clear, and with Microsoft pushing both, it’s a bit confusing to decide which one to use. Thanks!
That's a great question! Autogen is a framework for letting multiple agents collaborate with each other to complete a request. Semantic Kernel, on the other hand, is an SDK that helps you give a single agent a set of tools (via plugins). Because of this, we believe both projects complement each other. To make a team of agents in Autogen productive, you'll ultimately want to give them plugins so they can complete real work. This...
I was wondering the same question asked by @Bilal. I'm not sure I follow the response, specifically why we need Semantic Kernel to work with tools, as AutoGen also supports function-calling (tools/plugins) and it's used in most of their example use cases. Is there some difference in how AutoGen and Semantic Kernel approach functions/tools/plugins that should guide whether we use one or the other or both? Or are there functions/tools/plugins that cannot (easily) be created...
Thank you for the clarification, Matthew. So, if I've got this right, the aim of AutoGen is to enable collaboration among multiple LLMs/Agents (we use Azure OpenAI), while Semantic Kernel is designed to enhance the functionality and perform the integration with those agents using plugins. Given that we're currently using only one LLM, could we just stick with Semantic Kernel without the need to invest into AutoGen?
On another note, I've noticed in the SK main...
Even though y'all just have one LLM, you may see improvements by simulating a conversation between multiple agents with your single LLM. If so, you may want to try out AutoGen (but feel free to stick with Semantic Kernel if you only want a single agent).
Regarding connectors and plugins. In the future, the only connectors we'll be supporting are those for AI Services. All other features (calling databases, requesting memories, performing actions) will be described...
Thanks a lot Matthew for the explanation and suggestions. We will for sure give AutoGen a try!