AutoGen Agents Meet Semantic Kernel

John Maeda

In this blog post, we show you how you can use Semantic Kernel with AutoGen, a Microsoft Research project that shows the potential of using multiple agents together. With AutoGen, Microsoft research has shown that multiple agents can be better than one.

To actually do work, however, agents within AutoGen need hooks into the real world so they can retrieve information, create content, and complete tasks. Without these capabilities, agents will simply talk to each other and waste thousands or tens-of-thousands of tokens.

This is where Semantic Kernel comes in. With Semantic Kernel, you can easily bring Plugins to agents so they can do more than just talk. They can search the web, access company resources, and ultimately become productive parts of your company.

Plugins are important in the world of “agents.” Without them, it’s impossible to get agents to complete real work.

Why would you want to work with agents? Used improperly, they can quickly consume thousands of tokens, but given the right tools (i.e., plugins) and a constrained enough task, multiple agents powered by gpt-3.5-turbo can rival the capabilities of gpt-4, all while using cheaper tokens with lower latency. While it’s still early days for multi-agent systems, here at Microsoft we believe systems like AutoGen will substantially push the AI frontier forward.

How Agents Work

Agents are based on the principle that diverse teams produce better outcomes. Greater diversity in opinion is facilitated by essentially making LLMs chat with each other. In doing so, they go beyond a “Chain of Thought”-style monologue and instead involve seeding conversations between LLMs that have different Personas.

“In addition to diversity, there’s also specialization and modularity. Agents allow us to define AI copilots as specialized (i.e. “skilled agents”) on given goals, e.g. working on a specific area, looking after security, moderating content, etc. Better yet, agents can be combined. Modularity is important because it allows agent developers to focus on one area, rather than trying to create one do-it-all AI.” —Devis Lucato, SK Founding Architect

For example, imagine one agent with a copywriter system prompt:

const string SystemMessageCopyWriter = @"
You are a copywriter with ten years of experience and are known for a brevity and a dry humor.
You are laser-focused on the goal at hand. Don't waste time with chit chat.
The goal is to refine and decide on the best copy as an expert in the field.";

Kernel kernel = Kernel.CreateBuilder().AddAzureOpenAIChatCompletion(
                    MyModelId, MyEndpoint, MyApiKey)
                .Build();

var chatCompletion = kernel.GetRequiredService<IChatCompletionService>();

var chatHistory = new ChatHistory(SystemMessageCopyWriter);

// Add the user query to the chat history
chatHistory.AddUserMessage(userMessage);

// Finally, get the response from AI
var answer = await chatCompletion.GetChatMessageContentAsync(chatHistory);

Meanwhile, another agent can be given a different persona: an art director.

var systemMessageArtDirector = "You are an art director who has opinions about copywriting
born of a love for David Ogilvy. You're laser focused on the goal at hand.
Don't waste time with chit chat. The goal is to refine and decide on the best
copy as an expert in the field."

Using either Autogen or Semantic Kernel, you can then get these two agents to converse with each other. For example, the following conversation was generated by the two agents after giving them the initial marketing prompt: “maps made out of egg cartons.”

Copywriter: “Crack open a world of sustainability with our egg-carton maps. Navigate the globe, one recycled egg carton at a time.”

Art Director: The concept is intriguing, but the copy could be more direct and impactful. Let’s try: “Navigate the globe sustainably with our recycled egg-carton maps. One carton, one step towards a greener world.” This version emphasizes the product’s unique selling point and its environmental benefits more clearly.

Copywriter: “Recycled Egg-Carton Maps: Chart a greener course for our planet, one carton at a time.”

Art Director: This is a strong revision. It’s concise, direct, and clearly communicates the product’s unique selling point and environmental benefits. The phrase “Chart a greener course for our planet” is particularly effective, as it evokes a sense of adventure and responsibility. Well done.

Copywriter: Thank you. I’m glad you found the revised copy effective. If there’s anything else you need assistance with, feel free to ask.

Art Director: You’re welcome. Your revised copy is indeed effective. Keep up the good work. If you need further assistance, don’t hesitate to ask.

Using Dall-E2, the Art Director goes off and sketches up an illustration to accompany the copy:

Recycled Egg-Carton Maps: Chart a greener course for our planet, one carton at a time.

This is definitely destined for a Cannes Lion advertising industry award. Right?

More productive AutoGen agents With Semantic Kernel

AutoGen is a framework that enables the development of LLM applications using multiple agents that can converse with each other to solve tasks. AutoGen agents are customizable, conversable, and seamlessly allow human participation. They can operate in various modes that employ combinations of LLMs, human inputs, and tools.

Developing and finding the right tools to give AutoGen agents can be challenging though. Since this is an area where Semantic Kernel excels at, we wanted to demonstrate how you could give tools (i.e., plugins) to AutoGen that have already been imported into Semantic Kernel. These plugins could be authored as a collection of native functions or you could import an existing OpenAI plugin.

In our sample, we started by first creating two Semantic Kernel Plugins:

1. A Bing Search Plugin, used to retrieve from Bing URLs for a given topic.
2. A WebPages Plugin, used to fetch a given URL and return the text found.

We then show how you can use these two Semantic Kernel plugins within the world of AutoGen. By giving an AutoGen agent access to these plugins, the AutoGen agent can pull real-time information to answer user questions. And, because they’re the only plugins we gave it, we can ensure its working within the safe boundaries provided by the kernel.

Afterwards, a user can interact with AutoGen and it will use the plugins to:

1. Search for an answer using Bing,
2. Open a web page to get further details,
3. Summarize the results as an answer back to the user.

Please be sure to ⭐️ (star) the GitHub repo!

Interested in learning more about agents?

There’s a vintage piece written back in 2016 for Microsoft Developer Network Magazine that describes a multi-agent approach to using AI and simulating a basketball game. If you’re looking for something more up to date, check out Lilian Weng’s piece on autonomous agents to get a sense of where this field is heading today.

John Maeda

Follow