{"id":2138,"date":"2024-03-18T13:04:03","date_gmt":"2024-03-18T20:04:03","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/semantic-kernel\/?p=2138"},"modified":"2024-03-29T12:29:32","modified_gmt":"2024-03-29T19:29:32","slug":"guest-blog-building-your-custom-copilot-with-semantic-kernel","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/agent-framework\/guest-blog-building-your-custom-copilot-with-semantic-kernel\/","title":{"rendered":"Guest Blog: Building Your Custom Copilot with Semantic Kernel"},"content":{"rendered":"<p>Hello all, today we\u2019re featuring a guest on our Semantic Kernel blog. Arafat Tehsin, an MVP for Microsoft focused on AI. Arafat has been working with Global SIs of Microsoft and developing some next-gen solutions using Azure AI and .NET with the empowerment of our business users by Power Platform and more.<\/p>\n<p>We will turn it over to Arafat to dive into what he\u2019s been working on.<\/p>\n<p>Arafat: Hello\ud83d\udc4b. May I request you to write this prompt into your ChatGPT console and see what do you get?<\/p>\n<p style=\"padding-left: 40px;\">My friend is coming from Thailand. Can you please track the flight from Bangkok to Sydney? Also, considering the weather right now, can you suggest me some good thai food places?<\/p>\n<p>The response you will get is going to be similar to what I got here.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/ChatGPT-Prompt.png\"><img decoding=\"async\" class=\"alignnone wp-image-2166 size-large\" src=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/ChatGPT-Prompt-1024x664.png\" alt=\"Image ChatGPT Prompt\" width=\"640\" height=\"415\" srcset=\"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/ChatGPT-Prompt-1024x664.png 1024w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/ChatGPT-Prompt-300x195.png 300w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/ChatGPT-Prompt-768x498.png 768w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/ChatGPT-Prompt.png 1107w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p>It\u2019s because free version of ChatGPT is not connected to the internet. If you have got a ChatGPT Plus subscription or if you want to use Microsoft Copilot or Google Gemini you will get much better results. This is good for yourself but may not be good for your customers. Why? Because you do not have any control on this, as when and what services to call. Let\u2019s tackle these problems ourselves and create something interesting for our businesses and customers.<\/p>\n<p>This is the end goal we&#8217;re working to build:<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/Copilot-Prompt.png\"><img decoding=\"async\" class=\"alignnone wp-image-2144 size-large\" src=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/Copilot-Prompt-1024x571.png\" alt=\"Image Copilot Prompt\" width=\"640\" height=\"357\" srcset=\"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Copilot-Prompt-1024x571.png 1024w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Copilot-Prompt-300x167.png 300w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Copilot-Prompt-768x428.png 768w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Copilot-Prompt.png 1107w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p><strong>What are we doing?\u00a0<\/strong><\/p>\n<p>This post focuses on creating your own Copilot with\u00a0<a href=\"https:\/\/learn.microsoft.com\/semantic-kernel\/overview\/?WT.mc_id=AI-MVP-5003464\">Semantic Kernel<\/a>\u00a0powered by\u00a0<a href=\"https:\/\/learn.microsoft.com\/azure\/ai-services\/openai\/overview?WT.mc_id=AI-MVP-5003464\">Azure OpenAI Service<\/a>. We will try to leverage the strength of Large Language Models (LLMs) with the integration of external services. This will give you an idea of how you can really achieve your Copilot goals with not just with retail but with any industry, be it Power &amp; Utilities, Government and Public Sector and so on. I am deliberately refraining from using the term \u2018chatbot\u2019 as opposed to Copilot because of its overall capability and potential use-cases.<\/p>\n<p><strong>Vanilla Copilot<\/strong><\/p>\n<p>Let\u2019s start with building a basic chat client with Semantic Kernel. This is simple because all you have to do is to create a kernel with\u00a0AddAzureOpenAIChatCompletion\u00a0class and provide a system prompt with some boilerplate code. After providing the endpoint and API key (which I stored in my Environment Variables), you build the kernel, provide a system prompt with a loop that calls Azure OpenAI Service with streaming. This loop does not break because it\u2019s just a never-ending chatbot. However, you may want to build your own logic around that as when you want to stop it.<\/p>\n<pre class=\"prettyprint language-cs language-csharp\"><code class=\"language-cs language-csharp\">using Microsoft.SemanticKernel;\r\nusing Microsoft.SemanticKernel.ChatCompletion;\r\nusing Microsoft.SemanticKernel.Connectors.OpenAI;\r\nusing static System.Environment;\r\n\r\nnamespace CustomCopilot\r\n{\r\n    internal class Program\r\n    {\r\n\r\n        static async Task Main(string[] args)\r\n\r\n        {\r\n            \/\/ Create a kernel with the Azure OpenAI chat completion service\r\n            var builder = Kernel.CreateBuilder();\r\n            builder.AddAzureOpenAIChatCompletion(\"gpt-35-turbo-16k\",\r\n                GetEnvironmentVariable(\"AOI_ENDPOINT_SWDN\")!,\r\n                GetEnvironmentVariable(\"AOI_KEY_SWDN\")!);\r\n\r\n            \/\/ Build the kernel\r\n            var kernel = builder.Build();\r\n\r\n            \/\/ Create chat history\r\n            ChatHistory history = [];\r\n            history.AddSystemMessage(@\"You're a virtual assistant that helps people find information.\");\r\n\r\n            \/\/ Get chat completion service\r\n            var chatCompletionService = kernel.GetRequiredService&lt;IChatCompletionService&gt;();\r\n\r\n            \/\/ Start the conversation\r\n            while (true)\r\n            {\r\n                \/\/ Get user input\r\n                Console.ForegroundColor = ConsoleColor.White;\r\n                Console.Write(\"User &gt; \");\r\n                history.AddUserMessage(Console.ReadLine()!);\r\n\r\n                \/\/ Enable auto function calling\r\n                OpenAIPromptExecutionSettings openAIPromptExecutionSettings = new()\r\n                {\r\n                    MaxTokens = 200\r\n                };\r\n\r\n                \/\/ Get the response from the AI\r\n                var response = chatCompletionService.GetStreamingChatMessageContentsAsync(\r\n                               history,\r\n                               executionSettings: openAIPromptExecutionSettings,\r\n                               kernel: kernel);\r\n\r\n\r\n                Console.ForegroundColor = ConsoleColor.Green;\r\n                Console.Write(\"\\nAssistant &gt; \");\r\n\r\n                string combinedResponse = string.Empty;\r\n                await foreach (var message in response)\r\n                {\r\n                    \/\/Write the response to the console\r\n                    Console.Write(message);\r\n                    combinedResponse += message;\r\n                }\r\n\r\n                Console.WriteLine();\r\n\r\n                \/\/ Add the message from the agent to the chat history\r\n                history.AddAssistantMessage(combinedResponse);\r\n            }\r\n        }\r\n    }\r\n}<\/code><\/pre>\n<p>Now if you just take this file, replace it with your own\u00a0<code>Program.cs<\/code> in the Console Application and provide necessary configuration, it should just work like ChatGPT<\/p>\n<p>So, until now, it\u2019s the same demo(s) you may have seen on the internet to show you how you can use Azure OpenAI Service (or broadly OpenAI) API and enhance your business capability. Let\u2019s add some colors to it. Ask the same Copilot (i.e. ChatGPT clone),\u00a0<em>What\u2019s the time right now?<\/em><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/Output.png\"><img decoding=\"async\" class=\"alignnone wp-image-2173 size-full\" src=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/Output.png\" alt=\"Image Output\" width=\"867\" height=\"305\" srcset=\"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Output.png 867w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Output-300x106.png 300w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Output-768x270.png 768w\" sizes=\"(max-width: 867px) 100vw, 867px\" \/><\/a><\/p>\n<p>This is where things are going to start interesting. Did you know that the current enterprise LLMs are bad at telling you times, solving complex mathematical problems and keeping you posted with the facts (<em>simply because they are not connected to the internet, by default).<\/em><\/p>\n<p><strong>Plugins to the rescue<\/strong><\/p>\n<p><a href=\"https:\/\/arafattehsin.com\/build-your-ai-first-apps-with-semantic-kernel\/\">In my recent post<\/a>, I have already explained the necessary concepts and features of Semantic Kernel such as AI Plugins, Native Functions, Prompts and so on. Therefore, I won\u2019t be diving into too much of a detail, but I will just briefly explain here. <em>\u00a0<\/em><\/p>\n<p>Plugins are a way to add functionality and capabilities to your AI Copilots (or Agents). They are now based on the OpenAI plugin specification\u00a0<em>(thanks to the awesome SK team),<\/em>\u00a0which means they can interoperate with other AI platforms like ChatGPT, Microsoft 365 and so on.<em>\u00a0<\/em>Plugins can contain both native code (such as C# or Python) and prompts (such as requests to AI services). You can use plugins to access data, perform operations or augment your agent with other AI models. For example, you can create a plugin that can search the web, send emails, generate images or use even Microsoft Word\u00a0<em>(You can\u2019t just do this with the chatbot, can you?).<\/em>\u00a0Plugins are like the \u201carms and hands\u201d of your AI app, allowing it to interact with the real world.<\/p>\n<p><strong>Out of the box plugins<\/strong><\/p>\n<p>Another great thing about Semantic Kernel plugins is that some of them have been shipped by the team as a separate package. This means, you do not have to write all of your own, rather you can utilise some of the built-in ones too. For example, in our case, we\u2019ll be using the\u00a0TimePlugin\u00a0to solve this problem.<\/p>\n<p>All you have to do is to write this line of code and you\u2019re covered. This is extremely simple, easy and pluggable (hence the name, plugin).<\/p>\n<pre class=\"prettyprint language-cs language-csharp\"><code class=\"language-cs language-csharp\">\/\/ Create a kernel with the Azure OpenAI chat completion service\r\n var builder = Kernel.CreateBuilder();\r\n builder.AddAzureOpenAIChatCompletion(\"gpt-35-turbo-16k\",\r\n     GetEnvironmentVariable(\"AOI_ENDPOINT_SWDN\")!,\r\n     GetEnvironmentVariable(\"AOI_KEY_SWDN\")!);\r\n\r\n \/\/ Load the plugins &lt;br&gt; \/\/ add this namespace to use OOB plugins Microsoft.SemanticKernel.Plugins.Core;\r\n #pragma warning disable SKEXP0050\r\n builder.Plugins.AddFromType&lt;TimePlugin&gt;();\r\n \r\n \/\/ Build the kernel\r\n var kernel = builder.Build();<\/code><\/pre>\n<p><strong>Automatic Function Calling<\/strong><\/p>\n<p><strong>Quick note:<\/strong> It\u2019s worth mentioning that to use parallel function calling that you should be using the November version or later of OpenAI and vision models do <em>not<\/em> support function calling. Here is the full list of models that support function calling: <a href=\"https:\/\/nam06.safelinks.protection.outlook.com\/?url=https%3A%2F%2Fplatform.openai.com%2Fdocs%2Fguides%2Ffunction-calling%2Fsupported-models&amp;data=05%7C02%7Csopand%40microsoft.com%7C104b2f85d6424c8fc4e108dc50233a04%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638473359478956010%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=S0rDqmRtH8mNvRsr2Kw4PXz30apZ%2BBpu7NFbDXG0hZs%3D&amp;reserved=0\">https:\/\/platform.openai.com\/docs\/guides\/function-calling\/supported-models<\/a><\/p>\n<p>After adding the above code, here comes a little challenge for the Kernel. How would it know that I need to run\u00a0TimePlugin\u00a0when user asks about Time vs. the usual ChatGPT response? Broadly, there are two ways to handle this challenge.<\/p>\n<ol>\n<li>Get all the functions you registered yourself with the Plugin (in our case,\u00a0TimePlugin) and invoke the Function yourself (<em>I have slightly covered this process\u00a0<\/em><a href=\"https:\/\/arafattehsin.com\/build-your-ai-first-apps-with-semantic-kernel\/\"><em>here<\/em><\/a>) manually.<\/li>\n<li>Use auto function invocation feature of the Kernel (<em>super cool and you will know why!)<\/em><\/li>\n<\/ol>\n<p>So, we\u2019ll be going with the auto function calling and to enable that, all you need is a single line of code. Inside your while loop, you just have to set a\u00a0ToolCallBehavior\u00a0property.<\/p>\n<pre class=\"prettyprint language-cs language-csharp\"><code class=\"language-cs language-csharp\">\/\/ Get user input\r\nConsole.ForegroundColor = ConsoleColor.White;\r\nConsole.Write(\"User &gt; \");\r\nhistory.AddUserMessage(Console.ReadLine()!);\r\n\r\n\/\/ Enable auto function calling\r\nOpenAIPromptExecutionSettings openAIPromptExecutionSettings = new()\r\n{\r\n    MaxTokens = 200,\r\n    ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions\r\n};<\/code><\/pre>\n<p>After you add\u00a0ToolCallBehavior\u00a0and also add the\u00a0TimePlugin, you can run your app and should see the output like this.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/OutputwithTime-GPT.png\"><img decoding=\"async\" class=\"alignnone wp-image-2159 size-full\" src=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/OutputwithTime-GPT.png\" alt=\"Image OutputwithTime GPT\" width=\"652\" height=\"207\" srcset=\"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/OutputwithTime-GPT.png 652w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/OutputwithTime-GPT-300x95.png 300w\" sizes=\"(max-width: 652px) 100vw, 652px\" \/><\/a><\/p>\n<p>This works as expected. However, our goal is to create a\u00a0<strong>Retail Copilot<\/strong>. A copilot where we will add 3 distinct features on top of the strength of LLMs.<\/p>\n<p><strong>Retail Copilot<\/strong><\/p>\n<p>For the context of our copilot, we\u2019ll add 3 features (or you can also say Plugins). Each plugin will have its own significance and will be independent. This means that it can be added or removed based upon our needs without modifying a lot of code. This pluggable nature of the Copilot keeps our solution clean and managed. Here are the feature details:<\/p>\n<ol>\n<li>Flight Tracking<\/li>\n<li>Weather Radar<\/li>\n<li>Retail Finder<\/li>\n<\/ol>\n<p>We will now create a Plugin for each feature. To create a plugin, you need to define a group of functions (or a single function) that can be exposed to AI apps and services.\u00a0<strong>Each function should have a semantic description that explains what it does, what its input and output are and what side effects it may have i.e. what\u2019s happening inside the function with its return value<\/strong>. This helps the AI understand how to use your plugin and plan the best way to achieve the user\u2019s goal. You can also specify a persona for your plugin, which defines how your plugin should behave and communicate with the user. For example, you can make your plugin snarky, careful or friendly.<\/p>\n<p><strong>Flight Tracker\u00a0<\/strong><\/p>\n<p>This plugin will be using\u00a0<a href=\"https:\/\/aviationstack.com\/\">Aviation Stack API<\/a>\u00a0(free plan) to get the details of the flight. For free version, there\u2019s a requirement that you either provide an\u00a0IATA flight code\u00a0(for example,\u00a0EK 414) or you provide the\u00a0IATA source name\u00a0(such as\u00a0DXB \/ LHE) and\u00a0IATA destination name. Initially, I thought we would have to write 2 functions, one is to get the\u00a0IATA names\u00a0for example SYD for Sydney as the API expects it. Otherwise, you won\u2019t be able to get the right response. Just check the below screenshot, you can see that I provided\u00a0dep_iata\u00a0as Dubai and\u00a0arr_iata\u00a0as Sydney. It did not return anything.\u00a0However, when I changed it to the actual codes, it returned the correct response. Check this out below:<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/Postman-FlightTracking.png\"><img decoding=\"async\" class=\"alignnone wp-image-2161 size-full\" src=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/Postman-FlightTracking.png\" alt=\"Image Postman FlightTracking\" width=\"746\" height=\"665\" srcset=\"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Postman-FlightTracking.png 746w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Postman-FlightTracking-300x267.png 300w\" sizes=\"(max-width: 746px) 100vw, 746px\" \/><\/a><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTracker-Postman-Full.png\"><img decoding=\"async\" class=\"alignnone wp-image-2172 size-full\" src=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTracker-Postman-Full.png\" alt=\"Image FlightTracker Postman Full\" width=\"869\" height=\"778\" srcset=\"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTracker-Postman-Full.png 869w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTracker-Postman-Full-300x269.png 300w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTracker-Postman-Full-768x688.png 768w\" sizes=\"(max-width: 869px) 100vw, 869px\" \/><\/a><\/p>\n<h3><span style=\"font-size: 12pt;\"><strong>My most favorite part: Semantic Descriptions<\/strong><\/span><\/h3>\n<p>It is evident that you can\u2019t really get any response from the API unless you provide the correct\u00a0<code>IATA<\/code>\u00a0code names but interestingly, that\u2019s not the case. With the power of Semantic Descriptions of the function, it takes the power of both LLMs as well as the code we write to bring you the best results. Here\u2019s what I wrote for Flight Tracker:<\/p>\n<pre class=\"prettyprint language-cs language-csharp\"><code class=\"language-cs language-csharp\">using Microsoft.SemanticKernel;\r\nusing System;\r\nusing System.Collections.Generic;\r\nusing System.ComponentModel;\r\nusing System.Linq;\r\nusing System.Text;\r\nusing System.Threading.Tasks;\r\n\r\nnamespace CustomCopilot.Plugins.FlightTrackerPlugin\r\n{\r\n    public class FlightTrackerPlugin(string apiKey)\r\n    {\r\n        readonly HttpClient client = new HttpClient();\r\n\r\n        [KernelFunction, Description(\"Tracks the flight status of a provided source and destination\")]\r\n        [return: Description(\"Flight details and status\")]\r\n        public async Task&lt;string&gt; TrackFlightAsync(\r\n        [Description(\"IATA code for the source location\")] string source,\r\n        [Description(\"IATA code for the designation location\")] string destination,\r\n        [Description(\"IATA code for the flight\")] string flightNumber,\r\n        [Description(\"Count of flights\")] int limit)\r\n        {\r\n            string url = $\"http:\/\/api.aviationstack.com\/v1\/flights?access_key={apiKey}&amp;dep_iata={source}&amp;arr_iata={destination}&amp;limit={limit}&amp;flight_iata={flightNumber}\";\r\n\r\n            HttpResponseMessage response = await client.GetAsync(url);\r\n            response.EnsureSuccessStatusCode();\r\n            string responseBody = await response.Content.ReadAsStringAsync();\r\n\r\n            return responseBody;\r\n        }\r\n\r\n    }\r\n}<\/code><\/pre>\n<p>This is just one class, one function. This means, a single plugin with just one native function. If you notice, from the definition to its parameters to its return value, I have provided\u00a0<strong>Description<\/strong>\u00a0for almost everything in this function. The reason for these semantic descriptions is that Semantic Kernel will effectively use AI to process the request. That\u2019s why, if you notice\u00a0<code>source<\/code>\u00a0and\u00a0<code>destination<\/code>, I have written\u00a0<code>IATA code for the source \/ destination location<\/code>\u00a0respectively. By just giving this description, without the need of calling a separate service to resolve the city name, in our example\u00a0<code>Sydney<\/code>, it can automatically convert it to\u00a0<code>SYD<\/code>.\u00a0<em>This is the amazing feature for me!<\/em><\/p>\n<p>Once you\u2019re done with your Plugin class, all you have to do is to go back to your Program.cs file and add this plugin right after the\u00a0<code>TimePlugin<\/code>. Here\u2019s the code snippet for this:<\/p>\n<pre class=\"prettyprint language-cs language-csharp\"><code class=\"language-cs language-csharp\">builder.Plugins.AddFromObject(new FlightTrackerPlugin(GetEnvironmentVariable(\"AVIATIONSTACK_KEY\")!), nameof(FlightTrackerPlugin));<\/code><\/pre>\n<p>After adding this line of code. My suggestion is to change the system prompt to\u00a0<code>You're a virtual assistant that helps people track flight and find information.<\/code>\u00a0Execute and see if you get the similar response to this:<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTimeTracker-NoResponse.png\"><img decoding=\"async\" class=\"alignnone wp-image-2169 size-full\" src=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTimeTracker-NoResponse.png\" alt=\"Image FlightTimeTracker NoResponse\" width=\"1106\" height=\"190\" srcset=\"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTimeTracker-NoResponse.png 1106w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTimeTracker-NoResponse-300x52.png 300w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTimeTracker-NoResponse-1024x176.png 1024w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTimeTracker-NoResponse-768x132.png 768w\" sizes=\"(max-width: 1106px) 100vw, 1106px\" \/><\/a><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTimeTracker-Response.png\"><img decoding=\"async\" class=\"alignnone wp-image-2170 size-full\" src=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTimeTracker-Response.png\" alt=\"Image FlightTimeTracker Response\" width=\"1106\" height=\"190\" srcset=\"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTimeTracker-Response.png 1106w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTimeTracker-Response-300x52.png 300w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTimeTracker-Response-1024x176.png 1024w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/FlightTimeTracker-Response-768x132.png 768w\" sizes=\"(max-width: 1106px) 100vw, 1106px\" \/><\/a><\/p>\n<p>Now you can easily witness that our Copilot has already started taking a shape by interacting with the right plugin and also giving us the expected responses.<\/p>\n<h3><span style=\"font-size: 12pt;\"><strong>Weather Radar<\/strong><\/span><\/h3>\n<p>Just like the\u00a0<code>Flight Tracker Plugin<\/code>, you will write the\u00a0<code>Weather Plugin<\/code>. This plugin will be using the\u00a0<a href=\"https:\/\/www.weatherapi.com\/\">Weather API<\/a>\u00a0which is also free and very easy to use. I created a separate class file where I created a single function to get the weather.<\/p>\n<pre class=\"prettyprint language-cs language-csharp\"><code class=\"language-cs language-csharp\">using Microsoft.SemanticKernel;\r\nusing System;\r\nusing System.Collections.Generic;\r\nusing System.ComponentModel;\r\nusing System.Linq;\r\nusing System.Text;\r\nusing System.Threading.Tasks;\r\n\r\nnamespace CustomCopilot.Plugins.WeatherPlugin\r\n{\r\n    public class WeatherPlugin(string apiKey)\r\n    {\r\n        HttpClient client = new HttpClient();\r\n\r\n        [KernelFunction, Description(\"Gets the weather details of a given location\")]\r\n        [return: Description(\"Weather details\")]\r\n        public async Task&lt;string&gt; GetWeatherAsync(\r\n        [Description(\"name of the location\")] string locationName)\r\n        {\r\n            string url = $\"http:\/\/api.weatherapi.com\/v1\/current.json?key={apiKey}&amp;q={locationName}&amp;aqi=no\";\r\n\r\n            HttpResponseMessage response = await client.GetAsync(url);\r\n            response.EnsureSuccessStatusCode();\r\n            string responseBody = await response.Content.ReadAsStringAsync();\r\n\r\n            return responseBody;\r\n        }\r\n    }\r\n}<\/code><\/pre>\n<p>Following the above practice, you will just have to inject it writing the below line of code:<\/p>\n<pre class=\"prettyprint language-cs language-csharp\"><code class=\"language-cs language-csharp\">builder.Plugins.AddFromObject(new WeatherPlugin(GetEnvironmentVariable(\"WEATHERAPI_KEY\")!), nameof(WeatherPlugin));<\/code><\/pre>\n<div class=\"enlighter-default enlighter-v-standard enlighter-t-enlighter enlighter-l-csharp enlighter-hover enlighter-linenumbers \">\n<div class=\"\"><span style=\"font-size: 1rem; text-align: var(--bs-body-text-align);\">Now if you run it again, you will be able to see the weather coming up. <\/span><i style=\"font-size: 1rem; text-align: var(--bs-body-text-align);\">This is because your Copilot is interacting with the external services now.<\/i><\/div>\n<div><a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/Weather-NoResponse.png\"><img decoding=\"async\" class=\"alignnone wp-image-2164 size-full\" src=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/Weather-NoResponse.png\" alt=\"Image Weather NoResponse\" width=\"1109\" height=\"235\" srcset=\"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Weather-NoResponse.png 1109w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Weather-NoResponse-300x64.png 300w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Weather-NoResponse-1024x217.png 1024w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Weather-NoResponse-768x163.png 768w\" sizes=\"(max-width: 1109px) 100vw, 1109px\" \/><\/a><\/div>\n<\/div>\n<div><\/div>\n<div class=\"enlighter-default enlighter-v-standard enlighter-t-enlighter enlighter-l-csharp enlighter-hover enlighter-linenumbers \">\n<div><a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/Weather-Response.png\"><img decoding=\"async\" class=\"alignnone wp-image-2165 size-full\" src=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/Weather-Response.png\" alt=\"Image Weather Response\" width=\"1109\" height=\"235\" srcset=\"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Weather-Response.png 1109w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Weather-Response-300x64.png 300w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Weather-Response-1024x217.png 1024w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/Weather-Response-768x163.png 768w\" sizes=\"(max-width: 1109px) 100vw, 1109px\" \/><\/a><\/div>\n<\/div>\n<h3><span style=\"font-size: 12pt;\"><strong>Retail Finder<\/strong><\/span><\/h3>\n<p>Similar to the above, we\u2019ll now be writing a\u00a0<code>Places Suggestion Plugin<\/code>. This plugin will be using the\u00a0<a href=\"https:\/\/learn.microsoft.com\/rest\/api\/maps\/?view=rest-maps-2023-06-01&amp;WT.mc_id=AI-MVP-5003464\">Azure Maps<\/a>\u00a0(or Bing Maps API) to locate our desired places. It is also worth noting that changing\u00a0<code>System Message will have a greater impact. So let's change your existing system message to this:\u00a0<\/code><\/p>\n<p><code>You're a virtual assistant responsible for only flight tracking, weather updates and finding out the right places within Australia after inquiring about the proximity or city. You should not talk anything outside of your scope. Your response should be very concise and to the point. For each correct answer, you will get some $10 from me as a reward. Be nice with people.\u00a0<\/code><\/p>\n<p>This is how your Plugin code should look like.<\/p>\n<pre class=\"prettyprint language-cs language-csharp\"><code class=\"language-cs language-csharp\">using Azure.Maps.Search.Models;\r\nusing Azure.Maps.Search;\r\nusing Azure;\r\nusing Microsoft.SemanticKernel;\r\nusing System;\r\nusing System.Collections.Generic;\r\nusing System.Linq;\r\nusing System.Text;\r\nusing System.Threading.Tasks;\r\nusing System.ComponentModel;\r\n\r\nnamespace CustomCopilot.Plugins.PlaceSuggestionsPlugin\r\n{\r\n    public class PlaceSuggestionsPlugin\r\n    {\r\n        MapsSearchClient client;\r\n        HttpClient httpClient = new HttpClient();\r\n        string APIKey;\r\n\r\n        public PlaceSuggestionsPlugin(string apiKey)\r\n        {\r\n            APIKey = apiKey;\r\n            AzureKeyCredential credential = new(apiKey);\r\n            client = new MapsSearchClient(credential);\r\n        }\r\n\r\n        [KernelFunction, Description(\"Gets the place suggestions for a given location\")]\r\n        [return: Description(\"Place suggestions\")]\r\n        public async Task&lt;string&gt; GetPlaceSuggestionsAsync(\r\n        [Description(\"type of the place\")] string placeType,\r\n        [Description(\"name of the location\")] string locationName)\r\n        {\r\n            var searchResult = await client.SearchAddressAsync(locationName);\r\n\r\n            if (searchResult?.Value?.Results.Count() == 0) { return null; }\r\n\r\n            SearchAddressResultItem locationDetails = searchResult!.Value.Results[0];\r\n\r\n            string url = @$\"https:\/\/atlas.microsoft.com\/search\/fuzzy\/json?api-version=1.0&amp;query={placeType}\r\n                    &amp;subscription-key={APIKey}\r\n                    &amp;lat={locationDetails.Position.Latitude}\r\n                    &amp;lon={locationDetails.Position.Longitude}\r\n                    &amp;countrySet=AU\r\n                    &amp;language=en-AU\";\r\n\r\n            HttpResponseMessage response = await httpClient.GetAsync(url);\r\n            response.EnsureSuccessStatusCode();\r\n            string responseBody = await response.Content.ReadAsStringAsync();\r\n\r\n            return responseBody;\r\n        }\r\n    }\r\n}<\/code><\/pre>\n<p>Now, you can just add this line of code to your existing Copilot code and it should just start working fine.<\/p>\n<div class=\"enlighter-default enlighter-v-standard enlighter-t-enlighter enlighter-l-csharp enlighter-hover enlighter-linenumbers \">\n<div class=\"enlighter-code\">\n<div class=\"enlighter\">\n<div class=\"\">\n<pre class=\"prettyprint language-cs language-csharp\"><code class=\"language-cs language-csharp\">builder.Plugins.AddFromObject(new PlaceSuggestionsPlugin(GetEnvironmentVariable(\"AZUREMAPS_SUBSCRIPTION_KEY\")!), nameof(PlaceSuggestionsPlugin));<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<p>Launch your console app again and now you should see something similar to this:<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/RetailFinder-No-Response.png\"><img decoding=\"async\" class=\"alignnone wp-image-2162 size-full\" src=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/RetailFinder-No-Response.png\" alt=\"Image RetailFinder No Response\" width=\"1103\" height=\"472\" srcset=\"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/RetailFinder-No-Response.png 1103w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/RetailFinder-No-Response-300x128.png 300w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/RetailFinder-No-Response-1024x438.png 1024w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/RetailFinder-No-Response-768x329.png 768w\" sizes=\"(max-width: 1103px) 100vw, 1103px\" \/><\/a><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/RetailFinder-Response.png\"><img decoding=\"async\" class=\"alignnone wp-image-2163 size-full\" src=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/RetailFinder-Response.png\" alt=\"Image RetailFinder Response\" width=\"1104\" height=\"472\" srcset=\"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/RetailFinder-Response.png 1104w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/RetailFinder-Response-300x128.png 300w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/RetailFinder-Response-1024x438.png 1024w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/RetailFinder-Response-768x328.png 768w\" sizes=\"(max-width: 1104px) 100vw, 1104px\" \/><\/a><\/p>\n<p><strong><span style=\"font-size: 12pt;\">Bringing it all together<\/span><\/strong><\/p>\n<p>By now, you must have seen individual cases of your Copilot and they just work fine with external sources. Now, let\u2019s combine all of them together and try to ask something that\u2019d call all of your custom Plugins as well as use the power of LLMs.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/FinalResult-Copilot.png\"><img decoding=\"async\" class=\"alignnone wp-image-2168 size-full\" src=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2024\/03\/FinalResult-Copilot.png\" alt=\"Image FinalResult Copilot\" width=\"1103\" height=\"675\" srcset=\"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/FinalResult-Copilot.png 1103w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/FinalResult-Copilot-300x184.png 300w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/FinalResult-Copilot-1024x627.png 1024w, https:\/\/devblogs.microsoft.com\/agent-framework\/wp-content\/uploads\/sites\/78\/2024\/03\/FinalResult-Copilot-768x470.png 768w\" sizes=\"(max-width: 1103px) 100vw, 1103px\" \/><\/a><\/p>\n<p>Congratulations! You\u2019ve built your Copilot which is lightweight, independent of various plugins and also can be surfaced at any channel using different frameworks. It would be great to know about your use-case and how you would like to use Semantic Kernel and Azure OpenAI Service.<\/p>\n<p>To conclude, what does it mean to create a custom retail copilot with Semantic Kernel and Azure OpenAI Service? It means to harness the power of AI to augment our human capabilities and enrich our experiences. It means to bridge the gap between natural and formal languages, between human and machine intelligence, between local and global knowledge. It means to explore the possibilities of AI-First apps that can interact with a variety of services and domains. I have tried my best to share my journey of building such a copilot using Semantic Kernel as a flexible, lightweight and expressive framework with Azure OpenAI Service as a scalable and robust platform.<\/p>\n<p>From the Semantic Kernel team, we want to thank Arafat for his time and sharing out the amazing work he\u2019s been doing! We\u2019re always interested in hearing from you. If you have feedback, questions or want to discuss further, feel free to reach out to us and the community on the<a href=\"https:\/\/github.com\/microsoft\/semantic-kernel\/discussions\">\u00a0discussion boards<\/a> on GitHub! We would also love your support, if you&#8217;ve enjoyed using Semantic Kernel, give us a star on <a href=\"https:\/\/github.com\/microsoft\/semantic-kernel\">GitHub<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hello all, today we\u2019re featuring a guest on our Semantic Kernel blog. Arafat Tehsin, an MVP for Microsoft focused on AI. Arafat has been working with Global SIs of Microsoft and developing some next-gen solutions using Azure AI and .NET with the empowerment of our business users by Power Platform and more. We will turn [&hellip;]<\/p>\n","protected":false},"author":149071,"featured_media":2302,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[36,37],"class_list":["post-2138","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-semantic-kernel","tag-copilot-semantic-kernel-customer-copilot","tag-mvp-guest-blog"],"acf":[],"blog_post_summary":"<p>Hello all, today we\u2019re featuring a guest on our Semantic Kernel blog. Arafat Tehsin, an MVP for Microsoft focused on AI. Arafat has been working with Global SIs of Microsoft and developing some next-gen solutions using Azure AI and .NET with the empowerment of our business users by Power Platform and more. We will turn [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/2138","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/users\/149071"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/comments?post=2138"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/2138\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media\/2302"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media?parent=2138"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/categories?post=2138"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/tags?post=2138"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}