{"id":2556,"date":"2024-04-25T09:37:09","date_gmt":"2024-04-25T16:37:09","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/semantic-kernel\/?p=2556"},"modified":"2024-04-25T09:37:09","modified_gmt":"2024-04-25T16:37:09","slug":"introducing-api-manifest-plugins-for-semantic-kernel-2","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/agent-framework\/introducing-api-manifest-plugins-for-semantic-kernel-2\/","title":{"rendered":"Introducing API Manifest Plugins for Semantic Kernel"},"content":{"rendered":"<div>\n<p>Hi all,<\/p>\n<p>Today we\u2019re featuring a guest author from another team in Microsoft on our Semantic Kernel blog. We will turn it over to Mustafa Zengin to dive into Introducing API Manifest Plugins for Semantic Kernel.<\/p>\n<p>Semantic Kernel allows developers to import plugins from OpenAPI documents. For large APIs, such as <a href=\"https:\/\/learn.microsoft.com\/en-us\/graph\/overview\">Microsoft Graph<\/a>, importing the entire OpenAPI document can be inefficient and sometimes ineffective as the function descriptions will exceed token limits when used with large language models. To address this issue, we developed a new plugin infrastructure for Semantic Kernel that uses API Manifest, which is a convenient format to describe plugins that refer to subset of APIs and package them together.<\/p>\n<h2>What is an API Manifest?<\/h2>\n<div>An API Manifest is a document that describes an application&#8217;s API dependencies. It contains links to API descriptions, lists which requests are being made by the application, and the requests&#8217; authorization requirements. A complete specification of API Manifest can be found <a href=\"https:\/\/www.ietf.org\/archive\/id\/draft-miller-api-manifest-01.html\">here<\/a>.<\/div>\n<div><\/div>\n<div>It also describes API based plugins for AI chat interfaces.<\/div>\n<div><\/div>\n<\/div>\n<div>\n<h2>API Manifest Plugins for Semantic Kernel<\/h2>\n<div>API Manifest plugins for Semantic Kernel are a way to generate Semantic Kernel functions from API Manifest files. Semantic Kernel functions can be called from <a href=\"https:\/\/platform.openai.com\/docs\/guides\/function-calling\">function calling<\/a> (or <a href=\"https:\/\/docs.anthropic.com\/claude\/docs\/tool-use\">tool using<\/a>) large language models (LLMs) in various <a href=\"https:\/\/github.blog\/2024-04-04-what-is-retrieval-augmented-generation-and-what-does-it-do-for-generative-ai\/\">Retrieval Augmented Generation (RAG)<\/a> scenarios.<\/div>\n<div>It has a few key features:<\/div>\n<ul>\n<li>Slices large API descriptions into only the parts that are needed for the task at hand.<\/li>\n<li>Provides a way of packaging dependencies by combining multiple API dependencies into a single plugin.<\/li>\n<li>Defines the authorization requirements for the API calls.<\/li>\n<li>In copilot systems where plugins are provided by 3rd party authors, it allows quick inspection of the API dependencies and their authorization requirements as opposed to inspecting the entire API description.<\/li>\n<\/ul>\n<div>We will demonstrate the use of API Manifest plugins with a sample scenario.<\/div>\n<div><\/div>\n<h2>Scenario<\/h2>\n<div>Alice receives an email message in Turkish from a friend from her Astronomy Club who invites her to observe the solar eclipse together. The email talks about how rare the event is by referencing the previous solar eclipse as well. Alice will create a Markdown table that contains the dates of current and previous eclipses and a link to the eclipse map on timeanddate.com. She will also create an HTML table that contains the same information so that she can use them in different contexts.<\/div>\n<div>Turkish email content:<\/div>\n<blockquote>\n<div>8 Nisan 2024 tarihinde bir g\u00fcne\u015f tutulmas\u0131 olacak, beraber izlemek ister misin? Bir \u00f6nceki tutulma 4 Aral\u0131k 2021 tarihinde olmu\u015ftu. Bunu ka\u00e7\u0131rmak istemezsin.<\/div>\n<\/blockquote>\n<div>English translation of the email content:<\/div>\n<blockquote>\n<div>There will be a solar eclipse on April 8, 2024, would you like to watch it together? The previous eclipse was on December 4, 2021. You don&#8217;t want to miss this one.<\/div>\n<\/blockquote>\n<h2><\/h2>\n<h2>What are the tools available for this task?<\/h2>\n<ul>\n<li>Graph API\n<ul>\n<li>to access email content.<\/li>\n<\/ul>\n<\/li>\n<li>LLM\n<ul>\n<li>to translate from Turkish to English<\/li>\n<li>to extract dates from the email<\/li>\n<li>to build timeanddate.com link using <a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/ai-services\/openai\/concepts\/advanced-prompt-engineering?pivots=programming-language-chat-completions#few-shot-learning\">few-shot prompting<\/a><\/li>\n<li>to create Markdown table<\/li>\n<li>to create corresponding HTML table<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<div>In the last step, Alice would like the output to be consistent. However, LLMs are subject to hallucination, which means that consistency between the Markdown and HTML output is not always guaranteed. Conveniently GitHub has a Markdown API that outputs HTML representation of Markdown and does so in a consistent manner. So she will add GitHub API into the mix.<\/div>\n<ul>\n<li>GitHub API\n<ul>\n<li>to get the HTML representation of the Markdown table<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h2>How can this scenario be implemented in Semantic Kernel?<\/h2>\n<p>Microsoft Graph API has 1000s of operations in its OpenAPI description. We need some form of slicing of the API description. We will also connect to GitHub API to get the HTML representation of the Markdown table. These calls will have different authorization requirements. Graph API will need delegated authentication with <code>Mail.Read<\/code> scope for <code>\/me\/messages<\/code> endpoint <a href=\"https:\/\/learn.microsoft.com\/en-us\/graph\/api\/user-list-messages?view=graph-rest-1.0&amp;tabs=http#permissions\">per Graph API docs<\/a>. GitHub will work without authorization. API Manifest plugin infrastructure in Semantic Kernel provides all this functionality with a simple interface.<\/p>\n<div><\/div>\n<div>We will use <a href=\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/migrating-from-the-sequential-and-stepwise-planners-to-the-new-handlebars-and-stepwise-planner\/\">function calling stepwise planner<\/a> from Semantic Kernel to ask LLM to execute the following plan:<\/div>\n<blockquote>\n<div>Get my last email, extract the current and previous dates of eclipses, and build the timeanddate.com eclipse map URL. Then create a Markdown table and get the HTML representation of the Markdown table and output these two tables.<\/div>\n<div><\/div>\n<div>Eclipse map URL is built using the following format: https:\/\/www.timeanddate.com\/eclipse\/map\/2024-april-8<\/div>\n<\/blockquote>\n<div>(Note: for the sake of simplicity, we used &#8220;get my last email&#8221;, however, you can use more complicated queries, such as &#8220;get emails from a specific sender&#8221;, etc. All Graph API filters for the corresponding API call will be available in the API Manifest plugin)<\/div>\n<h3>Prerequisites<\/h3>\n<ul>\n<li><a href=\"https:\/\/learn.microsoft.com\/en-us\/graph\/auth-register-app-v2\">Register an application with Microsoft Identity Platform<\/a><\/li>\n<li><a href=\"https:\/\/azure.microsoft.com\/en-us\/products\/ai-services\/openai-service\">Create a model deployment with Azure OpenAI service<\/a>\n<ul>\n<li>Semantic Kernel has connectors for different LLMs, including locally hosted models. For the sample here, we will be using Azure OpenAI Service.<\/li>\n<\/ul>\n<\/li>\n<li>No prerequisites for GitHub API as it is anonymous.<\/li>\n<\/ul>\n<h3>Configuration<\/h3>\n<div>Fill in the details from the Prerequisites step to the <code>appsettings.Developer.json<\/code> file below.<\/div>\n<div><\/div>\n<pre>{\r\n    \"AzureOpenAI\": {\r\n        \"ChatModelId\": \"\",\r\n        \"ChatDeploymentName\": \"\",\r\n        \"Endpoint\": \"\",\r\n        \"ApiKey\": \"\"\r\n    },\r\n    \"MsGraph\": {\r\n        \"ClientId\": \"\",\r\n        \"TenantId\": \"9188040d-6c67-4c5b-b112-36a304b66dad\", \/\/ MSA\/Consumer\/Personal tenant,  https:\/\/learn.microsoft.com\/azure\/active-directory\/develop\/accounts-overview\r\n        \"RedirectUri\": \"http:\/\/localhost\"\r\n    }\r\n}\r\n<\/pre>\n<div><\/div>\n<div>Please note that if you are using a school or work account, you will need the tenant ID specified in your app registration.<\/div>\n<h3>Package dependencies<\/h3>\n<pre>dotnet add package Microsoft.Extensions.Configuration.Json\r\ndotnet add package Microsoft.SemanticKernel.Planners.OpenAI --prerelease \r\ndotnet add package Microsoft.SemanticKernel.Plugins.MSGraph --prerelease\r\ndotnet add package Microsoft.SemanticKernel.Plugins.OpenAPI.Extensions --prerelease\r\n<\/pre>\n<\/div>\n<h2><\/h2>\n<h3>API Manifest file<\/h3>\n<p>Currently, you can author this file manually. In the future, <a href=\"https:\/\/learn.microsoft.com\/en-us\/openapi\/kiota\/overview\">Kiota<\/a>, our open-source API-client generation tool, will provide the functionality to generate this file from OpenAPI descriptions.<\/p>\n<pre>{\r\n    \"applicationName\": \"Message Processor Plugin\",\r\n    \"description\": \"This plugin accesses Microsoft Graph API to read emails and GitHub API for Markdown to HTML conversion.\",\r\n    \"publisher\": {\r\n        \"name\": \"Alice\",\r\n        \"contactEmail\": \"alice@contoso.com\"\r\n    },\r\n    \"apiDependencies\": {\r\n        \"microsoft.graph\": {\r\n            \"apiDescriptionUrl\": \"https:\/\/raw.githubusercontent.com\/microsoftgraph\/msgraph-metadata\/master\/openapi\/v1.0\/graphexplorer.yaml\",\r\n            \"requests\": [\r\n                {\r\n                    \"method\": \"Get\",\r\n                    \"uriTemplate\": \"\/me\/messages\"\r\n                }\r\n            ]\r\n        },\r\n        \"github\": {\r\n            \"apiDescriptionUrl\": \"https:\/\/raw.githubusercontent.com\/github\/rest-api-description\/main\/descriptions\/api.github.com\/api.github.com.yaml\",\r\n            \"requests\": [\r\n                {\r\n                    \"method\": \"POST\",\r\n                    \"uriTemplate\": \"\/markdown\"\r\n                }\r\n            ]\r\n        }\r\n    }\r\n}\r\n<\/pre>\n<div><\/div>\n<ul>\n<li><code>apiDependencies<\/code> section contains the API dependencies.<\/li>\n<li><code>microsoft.graph<\/code> and <code>github<\/code> are the names of the dependencies.<\/li>\n<li><code>apiDescriptionUrl<\/code> is the URL of the OpenAPI description.<\/li>\n<li><code>requests<\/code> section contains the requests that are being made by the application.<\/li>\n<\/ul>\n<h3>Sample Program<\/h3>\n<pre>using Microsoft.Extensions.Configuration;\r\nusing Microsoft.SemanticKernel;\r\nusing Microsoft.SemanticKernel.Planning;\r\nusing Microsoft.SemanticKernel.Plugins.MsGraph.Connectors.CredentialManagers;\r\nusing Microsoft.SemanticKernel.Plugins.OpenApi;\r\nusing Microsoft.SemanticKernel.Plugins.OpenApi.Extensions;\r\n\r\n\/\/ read config\r\nvar configuration = new ConfigurationBuilder()\r\n    .AddJsonFile(\"appsettings.Development.json\", optional: true, reloadOnChange: true)\r\n    .Build();\r\n\r\n\/\/ initialize kernel\r\nvar kernel = Kernel.CreateBuilder()\r\n    .AddAzureOpenAIChatCompletion(\r\n        deploymentName: configuration[\"AzureOpenAI:ChatDeploymentName\"],\r\n        endpoint: configuration[\"AzureOpenAI:Endpoint\"],\r\n        serviceId: \"AzureOpenAIChat\",\r\n        apiKey: configuration[\"AzureOpenAI:ApiKey\"],\r\n        modelId: configuration[\"AzureOpenAI:ChatModelId\"]).Build();\r\n\r\n\/\/ prefetch graph token\r\nLocalUserMSALCredentialManager credentialManager = await LocalUserMSALCredentialManager.CreateAsync().ConfigureAwait(false);\r\n\r\nvar token = await credentialManager.GetTokenAsync(\r\n    configuration[\"MSGraph:ClientId\"],\r\n    configuration[\"MSGraph:TenantId\"],\r\n    \/\/ API Manifest file captures required scopes. In the future, we have plans to improve integrations to automate this process\r\n    \/\/ so that you don't have to specify the scopes in your auth callback.\r\n    [\"Mail.Read\"],\r\n    new Uri(configuration[\"MSGraph:RedirectUri\"])).ConfigureAwait(false);\r\n\r\n\/\/ create graph auth callback to inject auth header\r\nAuthenticateRequestAsyncCallback? graphAuthCallback = (HttpRequestMessage requestMessage, CancellationToken _) =&gt; {\r\n    requestMessage.Headers.Add(\"Authorization\", $\"Bearer {token}\");\r\n    return Task.CompletedTask;\r\n};\r\n\r\n\/\/ specify auth callbacks for each API dependency\r\n\/\/ we don't need one for GitHub as markdown API doesn't require any auth\r\nvar apiManifestPluginParameters = new ApiManifestPluginParameters\r\n{\r\n    FunctionExecutionParameters = new ()\r\n    {\r\n        { \"microsoft.graph\", new OpenApiFunctionExecutionParameters(authCallback: graphAuthCallback) },\r\n        \/\/ no need for github authentication as markdown API does not need it\r\n    }\r\n};\r\n\r\n\/\/ import api manifest plugin\r\nKernelPlugin plugin =\r\nawait kernel.ImportPluginFromApiManifestAsync(\r\n    \"MessageProcessorPlugin\",                   \/\/ plugin name\r\n    \"MessageProcessorPlugin\/apimanifest.json\",  \/\/ path to api manifest file\r\n    apiManifestPluginParameters)\r\n    .ConfigureAwait(false);\r\n\r\n\/\/ set goal\r\nvar goal = @\"\r\nGet my last email, extract the current and previous dates of eclipses, and build the timeanddate eclipse map URL. Then create a Markdown table and get the HTML representation of the Markdown table and output these two tables.\r\n\r\nEclipse map URL is built using the following format: https:\/\/www.timeanddate.com\/eclipse\/map\/2024-april-8\r\n\";\r\n\r\n\/\/ create planner\r\nvar planner = new FunctionCallingStepwisePlanner(\r\n    new FunctionCallingStepwisePlannerOptions\r\n    {\r\n        MaxIterations = 10,\r\n        MaxTokens = 32000\r\n    }\r\n);\r\n\r\n\/\/ execute plan\r\nvar result = await planner.ExecuteAsync(kernel, goal);\r\n\r\n\/\/ output result\r\nConsole.WriteLine(result.FinalAnswer);\r\n<\/pre>\n<h3>Output in Markdown<\/h3>\n<pre>| Eclipse Date | Eclipse URL |\r\n|--------------|-------------|\r\n| April 8, 2024 | [View Map](https:\/\/www.timeanddate.com\/eclipse\/map\/2024-april-8) |\r\n| December 4, 2021 | [View Map](https:\/\/www.timeanddate.com\/eclipse\/map\/2021-december-4) |\r\n<\/pre>\n<h3>Output in HTML (consistent with Markdown)<\/h3>\n<div><code>&lt;table role=\"table\"&gt;<\/code><\/div>\n<div><code>&lt;thead&gt;<\/code><\/div>\n<div><code>&lt;tr&gt;<\/code><\/div>\n<div><code>&lt;th&gt;Eclipse Date&lt;\/th&gt;<\/code><\/div>\n<div><code>&lt;th&gt;Eclipse URL&lt;\/th&gt;<\/code><\/div>\n<div><code>&lt;\/tr&gt;<\/code><\/div>\n<div><code>&lt;\/thead&gt;<\/code><\/div>\n<div><code>&lt;tbody&gt;<\/code><\/div>\n<div><code>&lt;tr&gt;<\/code><\/div>\n<div><code>&lt;td&gt;April 8, 2024&lt;\/td&gt;<\/code><\/div>\n<div><code>&lt;td&gt;&lt;a href=\"https:\/\/www.timeanddate.com\/eclipse\/map\/2024-april-8\" rel=\"nofollow\"&gt;View Map&lt;\/a&gt;&lt;\/td&gt;<\/code><\/div>\n<div><code>&lt;\/tr&gt;<\/code><\/div>\n<div><code>&lt;tr&gt;<\/code><\/div>\n<div><code>&lt;td&gt;December 4, 2021&lt;\/td&gt;<\/code><\/div>\n<div><code>&lt;td&gt;&lt;a href=\"https:\/\/www.timeanddate.com\/eclipse\/map\/2021-december-4\" rel=\"nofollow\"&gt;View Map&lt;\/a&gt;&lt;\/td&gt;<\/code><\/div>\n<div><code>&lt;\/tr&gt;<\/code><\/div>\n<div><code>&lt;\/tbody&gt;<\/code><\/div>\n<div><code>&lt;\/table&gt;<\/code><\/div>\n<h3>Rendered Output<\/h3>\n<table role=\"table\">\n<thead>\n<tr>\n<th>Eclipse Date<\/th>\n<th>Eclipse URL<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>April 8, 2024<\/td>\n<td><a href=\"https:\/\/www.timeanddate.com\/eclipse\/map\/2024-april-8\" rel=\"nofollow\">View Map<\/a><\/td>\n<\/tr>\n<tr>\n<td>December 4, 2021<\/td>\n<td><a href=\"https:\/\/www.timeanddate.com\/eclipse\/map\/2021-december-4\" rel=\"nofollow\">View Map<\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<div><\/div>\n<div>As both formats are available in a consistent style in the output, they can be used in places where Markdown or HTML is used, including this blog post.<\/div>\n<div>After defining the plugin once, the same plugin can be used in other message processing scenarios:<\/div>\n<ul>\n<li>Extracting different types of information from email messages will be only a matter of changing the goal in natural language.<\/li>\n<li>Generated HTML output can be wired to an email sender with the help of another plugin or with an extension of this plugin with additional endpoints from Graph.<\/li>\n<li>Generated Markdown output can be backed up to GitHub as a gist.<\/li>\n<li>and on and on&#8230;<\/li>\n<\/ul>\n<h3>Follow-up Challenge<\/h3>\n<div>NASA has a public API that returns &#8220;Astronomy Picture of the Day&#8221;. Extend the above scenario to include the &#8220;Astronomy Picture of the Day&#8221; for both eclipses. You should be able to extend API manifest file to refer to NASA API and add authentication with API key. Because connecting the NASA API will be outside the responsibility of &#8220;MessageProcessorPlugin&#8221;, you may want to rename it to something else or create two different plugins with clear package boundaries in terms of their responsibilities. It really depends on how you want to package your plugins and group the API dependencies. API Manifest plugins in Semantic Kernel give you that flexibility.<\/div>\n<div><\/div>\n<p>We are looking forward to what you are going to build with API Manifest plugins. Please let us know your feedback!&#8221;<\/p>\n<p>From the Semantic Kernel team, we want to thank Mustafa for his time. We\u2019re always interested in hearing from you. If you have feedback, questions or want to discuss further, feel free to reach out to us and the community on the <a href=\"https:\/\/github.com\/microsoft\/semantic-kernel\/discussions\/categories\/general\">Semantic Kernel GitHub Discussion Channel<\/a>! We would also love your support, if you\u2019ve enjoyed using Semantic Kernel, give us a star on\u00a0<a href=\"https:\/\/github.com\/microsoft\/semantic-kernel\">GitHub<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hi all, Today we\u2019re featuring a guest author from another team in Microsoft on our Semantic Kernel blog. We will turn it over to Mustafa Zengin to dive into Introducing API Manifest Plugins for Semantic Kernel. Semantic Kernel allows developers to import plugins from OpenAPI documents. For large APIs, such as Microsoft Graph, importing the [&hellip;]<\/p>\n","protected":false},"author":149071,"featured_media":2302,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[17,1],"tags":[48,69,70,9],"class_list":["post-2556","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-announcements","category-semantic-kernel","tag-ai","tag-api","tag-api-manifest-plugin","tag-semantic-kernel"],"acf":[],"blog_post_summary":"<p>Hi all, Today we\u2019re featuring a guest author from another team in Microsoft on our Semantic Kernel blog. We will turn it over to Mustafa Zengin to dive into Introducing API Manifest Plugins for Semantic Kernel. Semantic Kernel allows developers to import plugins from OpenAPI documents. For large APIs, such as Microsoft Graph, importing the [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/2556","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/users\/149071"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/comments?post=2556"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/2556\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media\/2302"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media?parent=2556"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/categories?post=2556"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/tags?post=2556"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}