{"id":1004,"date":"2023-08-10T07:29:41","date_gmt":"2023-08-10T14:29:41","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/semantic-kernel\/?p=1004"},"modified":"2023-08-10T07:29:41","modified_gmt":"2023-08-10T14:29:41","slug":"ai-tooling-for-java-developers-with-sk","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/agent-framework\/ai-tooling-for-java-developers-with-sk\/","title":{"rendered":"AI tooling for Java developers with SK"},"content":{"rendered":"

Every system needs to be able to add AI to its workflow to empower the users to complete their task much faster.\u00a0 The Semantic Kernel team and community have been working hard to create a Java based kernel to support Java developers to unleash AI into their apps.\u00a0 I will walk you through this journey below.<\/p>\n

We will review the following concepts with examples:<\/p>\n

    \n
  1. Kernel and use of Inline Function.<\/li>\n
  2. Plugins and Functions loaded from directory.<\/li>\n
  3. Plugins and Functions as pipeline.<\/li>\n
  4. Giving memory to LLM, using local volatile memory and Azure Cognitive Search.<\/li>\n<\/ol>\n

    All the examples are on GitHub: https:\/\/github.com\/sohamda\/SemanticKernel-Basics<\/a><\/p>\n

     <\/p>\n

    Step #1: Create a Kernel<\/h2>\n

    The kernel<\/a>\u00a0orchestrates a user’s ask. To do so, the kernel runs a\u00a0pipeline \/ chain<\/a> that is defined by a developer. While the chain is run, a common context is provided by the kernel so data can be shared between functions.<\/p>\n

    First create a Kernel object:<\/p>\n

    client.azureopenai.key=XXYY1232\r\nclient.azureopenai.endpoint=https:\/\/abc.openai.azure.com\/\r\nclient.azureopenai.deploymentname=XXYYZZ<\/pre>\n
    You can get these details from the Azure portal (endpoint and key) and Azure OpenAI (AOAI) studio (deployment name).<\/p>\n
    This article uses deployment \u201cgpt-35-turbo\u201d language model.<\/p>\n
    How to create such resource in Azure, follow this > How-to – Create a resource and deploy a model using Azure OpenAI Service – Azure OpenAI | Microsoft Learn<\/a><\/p>\n
    Then add:<\/p>\n
    AzureOpenAISettings settings = new AzureOpenAISettings(SettingsMap.\r\ngetWithAdditional<\/em>(List.of<\/em>(\r\nnew File(\"src\/main\/resources\/conf.properties\"))));\r\n\r\nOpenAIAsyncClient client = new OpenAIClientBuilder().endpoint(settings.getEndpoint()).credential(new AzureKeyCredential(settings.getKey())).buildAsyncClient();\r\nTextCompletion textCompletion = SKBuilders.chatCompletion<\/em>().build(client, settings<\/em>().getDeploymentName());\r\nKernelConfig config = SKBuilders.kernelConfig<\/em>().addTextCompletionService(\"text-completion\", kernel -> textCompletion).build();\r\nKernel kernel = SKBuilders.kernel<\/em>().withKernelConfig(config).build();<\/pre>\n
     <\/p>\n
    Step #2: Add a Function<\/h2>\n
    Function? Yes \ud83d\ude0a<\/p>\n
    Think of this as a prompt you provide to ChatGPT to direct it to do something. The below example, shows, how you create a \u201creusable\u201d prompt, aka function, to instruct the LLM to summarize a given input.<\/p>\n
    String semanticFunctionInline = \"\"\"\r\n{{$input}}\r\n\r\nSummarize the content above in less than 140 characters.\r\n\"\"\";<\/pre>\n
     <\/p>\n
    Step#3: Instruct LLM to use this Function + Additional Settings<\/h2>\n
    Now before you invoke AOAI API, you need to add this prompt\/function to the call and some additional settings, such as maxTokens, temperature, top etc.<\/p>\n
    FYI<\/strong>: LLMs are non-deterministic, meaning, the output for the same input might be different every time you invoke them. The \u201ctemperature\u201d parameter controls that, so higher temperature is higher level of non-deterministic behavior and lower temperature is lower that behavior.<\/p>\n
    CompletionSKFunction summarizeFunction = SKBuilders\r\n.completionFunctions<\/em>(kernel)\r\n.createFunction(\r\nsemanticFunctionInline,\r\nnew PromptTemplateConfig.CompletionConfigBuilder()\r\n.maxTokens(100)\r\n.temperature(0.4)\r\n.topP(1)\r\n.build());\r\n\r\nlog<\/em>.debug(\"== Run the Kernel ==\");\r\nMono<SKContext> result = summarizeFunction.invokeAsync(TextToSummarize<\/em>);\r\n\r\nlog<\/em>.debug(\"== Result ==\");\r\nlog<\/em>.debug(result.block().getResult());<\/pre>\n
    \u201cTextToSummarize\u201d is the user input, a String which we want to summarize using the \u201csemanticFunctionInline\u201d function and LLM.<\/p>\n
    Here is an example –> InlineFunction.java<\/a><\/p>\n
    CONGRATULATIONS<\/strong><\/span>!! \ud83d\ude0a You created your first JAVA<\/strong> program to interact with LLM using Semantic Kernel.<\/p>\n
     <\/p>\n
    Step #4: Plugins and Functions<\/h2>\n
    Functions are reusable prompts with AOAI API settings which can be used when invoking LLMs. A collection of functions is called a Plugin.<\/p>\n
    A Function is represented by a \u201cskprompt.txt\u201d and optionally a \u201cconfig.json\u201d.<\/p>\n
    Functions and Plugins are organized in a directory structure and ideally placed in the classpath. In this article examples, the Plugins and Function are placed inside \u201c\/resources\u201d folder, but you can think of separating them into their own repo and releasing them into JARs and using them into your application code as a dependency, making them easy to use and also make them part of the DevOps workflow and reusable by multiple applications.<\/p>\n
    \"Image<\/a><\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
    Considering SummarizeSkill in resources folder, with multiple functions in it:<\/p>\n
    \"Image<\/a><\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
     <\/p>\n
    To use, Summarize function from above with Semantic Kernel.<\/p>\n
    ReadOnlyFunctionCollection skill = kernel.importSkillFromDirectory(\"SummarizeSkill\", \"src\/main\/resources\/Skills\", \"SummarizeSkill\");\r\n\r\nCompletionSKFunction summarizeFunction = skill.getFunction(\"Summarize\", CompletionSKFunction.class);\r\n\r\nlog<\/em>.debug(\"== Set Summarize Skill ==\");\r\nSKContext summarizeContext = SKBuilders.context<\/em>().build();\r\nsummarizeContext.setVariable(\"input\", TextToSummarize<\/em>);\r\n\r\nlog<\/em>.debug(\"== Run the Kernel ==\");\r\nMono<SKContext> result = summarizeFunction.invokeAsync(summarizeContext);<\/pre>\n
    The code above does the following:<\/p>\n
      \n
    1. Loads a Plugins (Summarize) from a directory in classpath.<\/li>\n
    2. Define the function (Summarize) which needs to be used.<\/li>\n
    3. Add the input to the function and set it in the context<\/li>\n
    4. Invoke AOAI API with the function.<\/li>\n<\/ol>\n
      Here is a starter example –> SummarizerAsPrompt.java<\/a><\/p>\n
       <\/p>\n
      Step #5: Plugin& Function as code<\/h2>\n
      Well, we all love the fact that we can represent Plugins and functions as a Java class. Then we can instantiate and use it in OOB way and reduce String manipulation while using them.<\/p>\n
      Say No More \ud83d\ude0a<\/p>\n
      Still not in main branch, but this submodule defines a few you can use –> semantickernel-plugin-core<\/a><\/p>\n
      You can also implement your own, using annotations @DefineSKFunction, @SKFunctionInputAttribute, @SKFunctionParameters<\/p>\n
      In the below example, we use an existing Plugin, Summarize and its function Summarize, so basically the same example as in Step #4 but this time with object references.<\/p>\n
      import com.microsoft.semantickernel.coreskills.ConversationSummarySkill;\r\n\r\n\r\nlog<\/em>.debug(\"== Load Conversation Summarizer Skill from plugins ==\");\r\nReadOnlyFunctionCollection conversationSummarySkill =\r\nkernel.importSkill(new ConversationSummarySkill(kernel), null);\r\n\r\nlog<\/em>.debug(\"== Run the Kernel ==\");\r\nMono<SKContext> summary = conversationSummarySkill\r\n.getFunction(\"SummarizeConversation\", SKFunction.class)\r\n.invokeAsync(ChatTranscript<\/em>);<\/pre>\n
      The above code does the following:<\/p>\n
        \n
      1. Instantiate ConversationSummarySkill object.<\/li>\n
      2. Define the function (SummarizeConversation) which needs to be used.<\/li>\n
      3. Add the input to the function and set it in the context<\/li>\n
      4. Invoke AOAI API with the function.<\/li>\n<\/ol>\n
        You can also use Plugins and Function for all sorts of tasks within an AI pipeline. Think of transformation, translation, executing business logic and others. So don\u2019t see Plugins and functions as only to invoke LLM, but also do some other tasks which are needed for next step in the pipeline or transformation of output from previous step.<\/p>\n
        Example code samples\u00a0 –> SummarizerAsPlugin.java<\/a> and SkillPipeline.java<\/a><\/p>\n
         <\/p>\n
        Step #6: Giving LLM some memory<\/h2>\n
        What\u2019s memory? Think of giving LLM information to support it when executing a (or a set) of Plugins. For example, you want to get a summary of your dental insurance details from a 100-page insurance document, here you might choose to send the whole document along with your query to the LLM, but every model in LLM has a limit on tokens what it can process with per request. To manage that token limitation, it is helpful if you can search the document first, get the relevant pages or texts out of it and then query the LLM with those selected pages\/texts. That way LLM can efficiently summarize the content and not run out of tokens per request. The \u201dselected pages\/texts\u201d in this example are Memories.<\/p>\n
        For this example, we will be using 2 Kernels, one with AI Services type Embedding and another one with TextCompletion.<\/p>\n
        In Azure AI Studio, you will also need 2 deployments.<\/p>\n
        \"Image<\/a><\/p>\n
         <\/p>\n
         <\/p>\n
         <\/p>\n
         <\/p>\n
         <\/p>\n
         <\/p>\n
         <\/p>\n
        Kernel with Embedding is instantiated as follows:<\/p>\n
        OpenAIAsyncClient openAIAsyncClient = openAIAsyncClient<\/em>();\r\n\r\nEmbeddingGeneration<String> textEmbeddingGenerationService =\r\nSKBuilders.textEmbeddingGenerationService<\/em>()\r\n.build(openAIAsyncClient, \"embedding\");\r\n\r\nKernel kernel = SKBuilders.kernel<\/em>()\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0.withDefaultAIService(textEmbeddingGenerationService)\r\n.withMemoryStorage(SKBuilders.memoryStore<\/em>().build())\r\n.build();<\/pre>\n
        The above example uses a volatile memory, a local simulation of a vector DB.<\/p>\n
        Another way is to use Azure Cognitive Search, then the kernel would be instantiated as follows:<\/p>\n
        Kernel kernel = SKBuilders.kernel<\/em>()\r\n.withDefaultAIService(textEmbeddingGenerationService)\r\n.withMemory(new AzureCognitiveSearchMemory(\"<ACS_ENDPOINT>\", \"<ACS_KEY>\"))\r\n.build();<\/pre>\n
         <\/p>\n
        After you store your data in the memory, you can search on it, which gives you back the relevant sections of the memory.<\/p>\n
        kernel.getMemory()\r\n.searchAsync(\u201c<COLLECTION_NAME>\u201d,\r\n\u201c<QUERY>\u201d, 2, 0.7, true);<\/pre>\n
         <\/p>\n
        Then use this search results, extract the memory as text and then run, for example, a Summarizer Plugin using another Kernel object which uses TextCompletion as AI Service.<\/p>\n
        List<MemoryQueryResult> relevantMems = relevantMemory.block();\r\n\r\nStringBuilder memory = new StringBuilder();\r\nrelevantMems.forEach(relevantMem -> memory.append(\"URL: \").append(relevantMem.getMetadata().getId())\r\n.append(\"Title: \").append(relevantMem.getMetadata().getDescription()));\r\n\r\nKernel kernel = kernel<\/em>();\r\nReadOnlyFunctionCollection conversationSummarySkill =\r\nkernel.importSkill(new ConversationSummarySkill(kernel), null);\r\n\r\nMono<SKContext> summary = conversationSummarySkill\r\n.getFunction(\"SummarizeConversation\", SKFunction.class)\r\n.invokeAsync(relevantMemory);\r\n\r\nlog<\/em>.debug(summary.block().getResult());<\/pre>\n
         <\/p>\n
        The results will be a summary of the query you initially executed on the Embedding kernel.<\/p>\n
        Example Code –> SKWithMemory.java<\/a> and SKWithCognitiveSearch.java<\/a><\/p>\n
         <\/p>\n
        Next Steps<\/h3>\n
        Get involved with the Java Kernel development by contributing<\/a> and following the public Java board<\/a> to see what you might want to pick up.<\/p>\n
        Join the Semantic Kernel community<\/a>\u00a0and let us know what you think.<\/p>\n","protected":false},"excerpt":{"rendered":"
        Every system needs to be able to add AI to its workflow to empower the users to complete their task much faster.\u00a0 The Semantic Kernel team and community have been working hard to create a Java based kernel to support Java developers to unleash AI into their apps.\u00a0 I will walk you through this journey […]<\/p>\n","protected":false},"author":125575,"featured_media":1026,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[24,1,19],"tags":[],"class_list":["post-1004","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-java","category-semantic-kernel","category-vector-database"],"acf":[],"blog_post_summary":"
        Every system needs to be able to add AI to its workflow to empower the users to complete their task much faster.\u00a0 The Semantic Kernel team and community have been working hard to create a Java based kernel to support Java developers to unleash AI into their apps.\u00a0 I will walk you through this journey […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/1004","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/users\/125575"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/comments?post=1004"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/1004\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media\/1026"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media?parent=1004"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/categories?post=1004"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/tags?post=1004"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}