{"id":3100,"date":"2024-08-01T15:13:19","date_gmt":"2024-08-01T22:13:19","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/semantic-kernel\/?p=3100"},"modified":"2024-08-20T09:10:16","modified_gmt":"2024-08-20T16:10:16","slug":"introducing-agents-in-semantic-kernel","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/agent-framework\/introducing-agents-in-semantic-kernel\/","title":{"rendered":"Introducing enterprise multi-agent support in Semantic Kernel"},"content":{"rendered":"<p>The term &#8220;agent&#8221; has become a popular term within the industry. There have <em>many<\/em> different definitions, but at their core, they consist of a system prompt (i.e., a <a href=\"https:\/\/learn.microsoft.com\/en-us\/semantic-kernel\/concepts\/personas?pivots=programming-language-csharp\">persona<\/a>), <a href=\"https:\/\/learn.microsoft.com\/en-us\/semantic-kernel\/concepts\/plugins\/?pivots=programming-language-csharp\">plugins<\/a>, and an ability to automatically reason and <a href=\"https:\/\/learn.microsoft.com\/en-us\/semantic-kernel\/concepts\/planning?pivots=programming-language-csharp\">create plans<\/a> to address user goals.<\/p>\n<p>Up until today, we&#8217;ve demonstrated how you could use components of Semantic Kernel to build agents. With just a few lines of code, you can use a <a href=\"https:\/\/learn.microsoft.com\/en-us\/semantic-kernel\/concepts\/ai-services\/chat-completion\/?tabs=csharp-AzureOpenAI%2Cpython-AzureOpenAI%2Cjava-AzureOpenAI&amp;pivots=programming-language-csharp\">chat completion model<\/a> to answer user&#8217;s questions and to automatically invoke plugins as necessary.<\/p>\n<p>What was missing, however, was a first-class agent abstraction. Not only would this simplify code by consolidating logic, but it would also ensure that there is a common contract on\u00a0<em>how<\/em> to interact with an agent. This may seem like a small feat, but it has allowed us to build a multi-agent framework that allows agents to coordinate with one another while also simplifying the code you need to write.<\/p>\n<p>You can find the agent packages for the Python and .NET SDKs in the following locations:<\/p>\n<ul>\n<li><a href=\"https:\/\/www.nuget.org\/packages\/Microsoft.SemanticKernel.Agents.Abstraction\">C#: Microsoft.SemanticKernel.Agents.Abstraction (<\/a><a href=\"https:\/\/www.nuget.org\/packages\/Microsoft.SemanticKernel.Agents.Abstraction\">nuget.org<\/a>)<\/li>\n<li><a href=\"https:\/\/pypi.org\/project\/semantic-kernel\/\">Python: PyPi package<\/a><\/li>\n<\/ul>\n<h2>Introducing Semantic Kernel agents<\/h2>\n<p>With our latest Python (1.6.0) and .NET releases (1.18.0 RC1), Semantic Kernel now provides a first-class abstraction for agents. This reduces much of the complexity required to build a standard chat experience while also providing a standardized API to interact with them. With this release, we&#8217;re providing two out-of-the-box agents: Assistant API agents and Chat completion agents.<\/p>\n<p><div class=\"alert alert-information\">Note: The <em>Agent Framework<\/em> is currently marked as experimental along with other certain <em>Semantic Kernel<\/em> features.\u00a0 Even though we strive to maintain stable development patterns, there may be updates that break compatibility until we are able to graduate the <em>Agent Framework<\/em>.<\/div><\/p>\n<h3>Assistant API agents<\/h3>\n<p>For most developers, the <span style=\"font-family: terminal, monaco, monospace;\">OpenAIAssistantAgent<\/span> (based on the <a href=\"https:\/\/platform.openai.com\/docs\/api-reference\/assistants\">Open AI Assistant API<\/a>) provides the simplest way to get started. With OpenAI Assistants, state is managed automatically for you and you&#8217;re provided out-of-the-box tools (i.e., code interpreter and file retrieval).<\/p>\n<p>To use Assistants in Semantic Kernel, you can use the Semantic Kernel <a href=\"https:\/\/pypi.org\/project\/semantic-kernel\/\">python package<\/a> or the <a href=\"https:\/\/www.nuget.org\/packages\/Microsoft.SemanticKernel.Agents.OpenAI\">Assistant agent .NET NuGet package<\/a>. Once you&#8217;ve installed and imported the right packages, you can quickly get started with just a few lines of code.<\/p>\n<p><em>Note: the following code samples are high-level to show the main building blocks for agents. Please follow the provided links, below the code snippets, to view the full implementation of each scenario.<\/em><\/p>\n<p><em><strong>C# code<\/strong><\/em><\/p>\n<pre><code>IKernelBuilder builder = Kernel.CreateBuilder();\r\n\r\nbuilder.Plugins.AddFromType&lt;YourPlugin&gt;();\r\n\r\nKernel kernel = builder.Build();\r\n\r\nOpenAIAssistantAgent agent =\r\n    await OpenAIAssistantAgent.CreateAsync(\r\n        kernel: new(),\r\n        config: new(YourConfiguration.AzureOpenAI.ApiKey, YourConfiguration.AzureOpenAI.Endpoint),\r\n        new()\r\n        {\r\n            Name = \"&lt;agent name&gt;\",\r\n            Instructions = \"&lt;agent instructions&gt;\",\r\n            ModelId = YourConfiguration.AzureOpenAI.ChatDeploymentName,\r\n            EnableCodeInterpreter = true,\r\n        });\r\n\r\nstring threadId = await agent.CreateThreadAsync();\r\n\r\nawait agent.AddChatMessageAsync(threadId, new ChatMessageContent(AuthorRole.User, \"&lt;input&gt;\"));\r\n\r\nawait foreach (ChatMessageContent message in agent.InvokeAsync(threadId))\r\n{\r\n    DisplayMessage(message);\r\n}\r\n<\/code><\/pre>\n<p>See the\u00a0C# <a href=\"https:\/\/github.com\/microsoft\/semantic-kernel\/blob\/main\/dotnet\/samples\/GettingStartedWithAgents\/Step8_OpenAIAssistant.cs\">Step8_OpenAIAssistant.cs<\/a> code sample for the complete implementation.<\/p>\n<p><em><strong>Python code<\/strong><\/em><\/p>\n<pre><code>kernel = Kernel()\r\n\r\nservice_id = \"agent\"\r\n\r\nkernel.add_plugin(plugin=YourPlugin(), plugin_name=\"your_plugin\")\r\n\r\nagent = await AzureAssistantAgent.create(\r\n    kernel=kernel, service_id=service_id, name=\"&lt;agent name&gt;\", instructions=\"&lt;agent instructions&gt;\"\r\n)\r\n\r\nthread_id = await agent.create_thread()\r\n\r\nawait agent.add_chat_message(thread_id=thread_id, message=ChatMessageContent(role=AuthorRole.USER, content=\"&lt;input&gt;\"))\r\n\r\nasync for message in agent.invoke(thread_id=thread_id):\r\n    display_message(message)\r\n<\/code><\/pre>\n<p>See the Python\u00a0<a href=\"https:\/\/github.com\/microsoft\/semantic-kernel\/blob\/main\/python\/samples\/getting_started_with_agents\/step7_assistant.py\">step7_assistant.py<\/a> code sample for the complete implementation.<\/p>\n<h3>Chat completion agent<\/h3>\n<p>For customers who don&#8217;t want to use Assistants (either because they want more control over their chat history or because they want to use a non-OpenAI model), they can use our Chat completion agent. Internally it behaves the same as OpenAI Assistants, but you need to provide your own chat history and must supply your own code interpreter.<\/p>\n<p>To use the chat completion agent in Semantic Kernel, you can use the Semantic Kernel <a href=\"https:\/\/pypi.org\/project\/semantic-kernel\/\">python package<\/a> or the .<a href=\"https:\/\/www.nuget.org\/packages\/Microsoft.SemanticKernel.Agents.Core\/\">Core agent .NET NuGet package<\/a>. Once you&#8217;ve installed and imported the right packages, you can quickly get started with just a few lines of code:<\/p>\n<p><em><strong>C# code<\/strong><\/em><\/p>\n<pre><code>IKernelBuilder builder = Kernel.CreateBuilder();\r\n\r\nbuilder.AddAzureOpenAIChatCompletion(\r\n    YourConfiguration.AzureOpenAI.ChatDeploymentName,\r\n    YourConfiguration.AzureOpenAI.Endpoint,\r\n    YourConfiguration.AzureOpenAI.ApiKey);\r\n\r\nbuilder.Plugins.AddFromType&lt;YourPlugin&gt;()\r\n\r\nKernel kernel = builder.Build();\r\n\r\nChatCompletionAgent agent =\r\n    new()\r\n    {\r\n        Name = \"&lt;agent name&gt;\",\r\n        Instructions = \"&lt;agent instructions&gt;\",\r\n        Kernel = kernel,\r\n    };\r\n\r\nChatHistory chat = [new ChatMessageContent(AuthorRole.User, \"&lt;input&gt;\")];\r\n\r\nawait foreach (ChatMessageContent message in agent.InvokeAsync(chat))\r\n{\r\n    chat.Add(message);\r\n    DisplayMessage(message);\r\n}\r\n<\/code><\/pre>\n<p>See the C#\u00a0<a href=\"https:\/\/github.com\/microsoft\/semantic-kernel\/blob\/main\/dotnet\/samples\/GettingStartedWithAgents\/Step1_Agent.cs\">Step1_Agent.cs<\/a> code sample for the complete implementation.<\/p>\n<p><em><strong>Python code<\/strong><\/em><\/p>\n<pre><code>kernel = Kernel()\r\n\r\nservice_id = \"agent\"\r\n\r\nkernel.add_service(AzureChatCompletion(service_id=service_id))\r\n\r\nkernel.add_plugin(plugin=YourPlugin(), plugin_name=\"your_plugin\")\r\n\r\nagent = ChatCompletionAgent(service_id=service_id, kernel=kernel, name=\"&lt;agent name&gt;\", instructions=\"&lt;agent instructions&gt;\")\r\n\r\nchat = ChatHistory()\r\n\r\nchat.add_user_message(\"&lt;input&gt;\")\r\n\r\nasync for content in agent.invoke(chat):\r\n    chat.add_message(content)\r\n    display_message(message)\r\n<\/code><\/pre>\n<p>See the Python\u00a0<a href=\"https:\/\/github.com\/microsoft\/semantic-kernel\/blob\/main\/python\/samples\/getting_started_with_agents\/step1_agent.py\">step1_agent.py<\/a> code sample for the complete implementation.<\/p>\n<h2>Multi-agent orchestration: group chat vs task-based<\/h2>\n<p>Once you have a single agent, you can then create multiple and have them work together, there&#8217;s been a considerable amount of research that has gone into multi-agent patterns over the past several months. Very early on, projects like AutoGen were able to identify that multiple agents, with different tools and system prompts, could outperform a single agent. This is no different than humans. With a diverse group of people, multiple perspectives and skillsets can be leveraged to deliver a better product or service.<\/p>\n<p>What has emerged are two different ways of coordinating (or orchestrating) multiple agents together: <strong>group chats<\/strong> or <strong>task-based business process flows<\/strong>.<\/p>\n<h3>Collaborating within a group chat<\/h3>\n<p>With group chats, several different participants (multiple agents and\/or multiple humans) can come together and share a common chat history. This pattern is no different than how humans use apps like Microsoft Teams, WhatsApp, or SMS. It&#8217;s best at preserving semantic information across multiple turns. For example, if a user told agent A that they liked the color blue, agent B would be able to use that same information because they&#8217;re in the same chat.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/08\/GroupChat.png\"><img decoding=\"async\" class=\"aligncenter wp-image-3169 size-medium\" src=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/08\/GroupChat-300x246.png\" alt=\"Image GroupChat\" width=\"300\" height=\"246\" srcset=\"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/08\/GroupChat-300x246.png 300w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/08\/GroupChat-1024x838.png 1024w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/08\/GroupChat-768x629.png 768w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/08\/GroupChat.png 1456w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>This pattern, however, isn&#8217;t perfect. Agents can get stuck in a loop talking back-and-forth with each other and it can require many tokens (thereby making solutions slow and costly). Because of this, another breed of orchestration was introduced by the research community&#8230;<\/p>\n<h3>Providing structure with business process flows (coming soon)<\/h3>\n<p>The other popular approach was influenced by how humans work together on complex business processes. When employees work on a large project, they don&#8217;t just use chat; the conversation would become unwieldy and overwhelming. Instead, other productivity tools are leveraged, like Microsoft 365 and GitHub. At the center of the work though, is typically some sort of task app that coordinates who is working on what and tracks the progress that is being made.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/08\/Process.png\"><img decoding=\"async\" class=\"aligncenter wp-image-3170 size-medium\" src=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/08\/Process-300x195.png\" alt=\"Image Process\" width=\"300\" height=\"195\" srcset=\"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/08\/Process-300x195.png 300w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/08\/Process-1024x667.png 1024w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/08\/Process-768x500.png 768w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/08\/Process-1536x1001.png 1536w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/08\/Process.png 1636w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>Frameworks like Crew AI were the first to build this pattern as a first-class piece of their framework. Instead of just passing conversation state from one agent to another, artifacts are routed between agents so that they can progressively complete work that is leveraged by someone else. Unfortunately,\u00a0<em>because<\/em> the primary means of communication is via artifacts, any information in a chat that isn&#8217;t included in an artifact is effectively lost.<\/p>\n<h3>The best of both worlds: combining group chats with processes<\/h3>\n<p>Most of today&#8217;s frameworks either do one or the other (not both) or interweave them into the same concept. What the Semantic Kernel team has identified though is that there&#8217;s benefit of treating them as separate concepts that can be interwoven together. We can take additional inspiration from how humans work. We don&#8217;t have monolithic apps that combine chat and tasks. No, we typically have a single enterprise chat application (e.g., Microsoft Teams) and a separate project management app (e.g., GitHub projects) to manage work.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/08\/processchat.png\"><img decoding=\"async\" class=\"aligncenter wp-image-3171 size-medium\" src=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/08\/processchat-300x276.png\" alt=\"Image process chat\" width=\"300\" height=\"276\" srcset=\"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/08\/processchat-300x276.png 300w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/08\/processchat-1024x941.png 1024w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/08\/processchat-768x706.png 768w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/08\/processchat-1536x1412.png 1536w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/08\/processchat.png 1636w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/a><\/p>\n<p>As we approach Microsoft Ignite, we&#8217;re going to deliver preview versions of both patterns and demonstrate how they can work together. <strong>Today, we&#8217;re exciting to share the .NET preview version of Group Chat orchestration.<\/strong><\/p>\n<h2>Introducing AutoGen group chats in Semantic Kernel<\/h2>\n<p>As a research project, AutoGen has not been developed to support enterprise production scenarios, but\u00a0<em>many<\/em> customers have expressed interest in using the patterns established with AutoGen in their AI applications. To support these customers, we&#8217;ve replicated most of the patterns in AutoGen within Semantic Kernel within the .NET library.<\/p>\n<blockquote><p>Development for group chats in Java is currently ongoing and is planned to be released by Ignite.<\/p><\/blockquote>\n<p>At the core of AutoGen and Semantic Kernel&#8217;s agent framework is an <span style=\"font-family: terminal, monaco, monospace;\">AgentChat<\/span>. This object stores the shared conversation state across multiple agents and also contains the routing logic to determine &#8220;who speaks next,&#8221; and when a conversation round is over. Below, you can see how easy it is in Semantic Kernel to create a group chat that has a custom termination strategy.<\/p>\n<p><em><strong>C# code<\/strong><\/em><\/p>\n<pre><code>ChatCompletionAgent agentReviewer = ...;\r\nOpenAIAssistantAgent agentWriter = ...;\r\n\r\nAgentGroupChat chat =\r\n    new(agentWriter, agentReviewer)\r\n    {\r\n        ExecutionSettings =\r\n            new()\r\n            {\r\n                TerminationStrategy =\r\n                    new ApprovalTerminationStrategy()\r\n                    {\r\n                        Agents = [agentReviewer]\r\n                    }\r\n            }\r\n    };\r\n\r\nchat.AddChatMessage(new ChatMessageContent(AuthorRole.User, \"&lt;input&gt;\"));\r\n\r\nawait foreach (ChatMessageContent content in chat.InvokeAsync())\r\n{\r\n    DisplayMessage(message);\r\n}\r\n<\/code><\/pre>\n<p>See the C#\u00a0<a href=\"https:\/\/github.com\/microsoft\/semantic-kernel\/blob\/main\/dotnet\/samples\/GettingStartedWithAgents\/Step3_Chat.cs\">Step3_Chat.cs<\/a> code sample for the complete implementation.<\/p>\n<p><em><strong>Python code<\/strong><\/em><\/p>\n<pre><code>agent_reviewer = ChatCompletionAgent(...)\r\nagent_writer = await OpenAIAssistantAgent.create(...)\r\n\r\nchat = AgentGroupChat(\r\n    agents=[agent_writer, agent_reviewer],\r\n    termination_strategy=ApprovalTerminationStrategy(agents=[agent_reviewer]),\r\n)\r\n\r\nawait chat.add_chat_message(ChatMessageContent(role=AuthorRole.USER, content=\"&lt;input&gt;\"))\r\n\r\nasync for message in chat.invoke():\r\n    display_message(message)\r\n<\/code><\/pre>\n<p>See the Python\u00a0<a href=\"https:\/\/github.com\/microsoft\/semantic-kernel\/blob\/main\/python\/samples\/getting_started_with_agents\/step3_chat.py\">step3_chat.py<\/a> code sample for the complete implementation.<\/p>\n<h2>What&#8217;s next?<\/h2>\n<p>We are continuing to evolve the <em>Agent Framework<\/em> and look forward to your input and suggestions. Besides delivering a system for process automation, we&#8217;re also working on providing&#8230;<\/p>\n<ul>\n<li>Support for Python OpenAI Assistant V2 features is available starting from version 1.4.0. Support for .NET OpenAI Assistant V2 features is available as of 1.18.0 RC1.<\/li>\n<li>Enable for serializing and restoring <span style=\"font-family: terminal, monaco, monospace;\">AgentChat<\/span> (soon!)<\/li>\n<li>Expose for streaming for all agents and <span style=\"font-family: terminal, monaco, monospace;\">AgentChat<\/span><\/li>\n<li>Enable history truncation strategy for <span style=\"font-family: terminal, monaco, monospace;\">ChatCompletionAgent<\/span><\/li>\n<li>Improved <em>chat<\/em> patterns<\/li>\n<\/ul>\n<p>Please reach out if you have any questions or feedback through our\u00a0<a href=\"https:\/\/github.com\/microsoft\/semantic-kernel\/discussions\/categories\/general\" target=\"_blank\" rel=\"noopener\">Semantic Kernel GitHub Discussion Channel<\/a>. We look forward to hearing from you!\u00a0We would also love your support, if you\u2019ve enjoyed using Semantic Kernel, give us a star on\u00a0<a href=\"https:\/\/github.com\/microsoft\/semantic-kernel\" target=\"_blank\" rel=\"noopener\">GitHub<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The term &#8220;agent&#8221; has become a popular term within the industry. There have many different definitions, but at their core, they consist of a system prompt (i.e., a persona), plugins, and an ability to automatically reason and create plans to address user goals. Up until today, we&#8217;ve demonstrated how you could use components of Semantic [&hellip;]<\/p>\n","protected":false},"author":124422,"featured_media":2364,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[79,88,48,49,63,53,9],"class_list":["post-3100","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-semantic-kernel","tag-net","tag-agents","tag-ai","tag-ai-agents","tag-microsoft-semantic-kernel","tag-python","tag-semantic-kernel"],"acf":[],"blog_post_summary":"<p>The term &#8220;agent&#8221; has become a popular term within the industry. There have many different definitions, but at their core, they consist of a system prompt (i.e., a persona), plugins, and an ability to automatically reason and create plans to address user goals. Up until today, we&#8217;ve demonstrated how you could use components of Semantic [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/3100","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/users\/124422"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/comments?post=3100"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/3100\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media\/2364"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media?parent=3100"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/categories?post=3100"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/tags?post=3100"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}