{"id":2523,"date":"2024-04-24T08:48:18","date_gmt":"2024-04-24T15:48:18","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/semantic-kernel\/?p=2523"},"modified":"2024-04-24T08:48:18","modified_gmt":"2024-04-24T15:48:18","slug":"unit-testing-with-semantic-kernel","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/agent-framework\/unit-testing-with-semantic-kernel\/","title":{"rendered":"Unit Testing with Semantic Kernel"},"content":{"rendered":"

Hi all,<\/p>\n

Testing is an integral part of the software development process. Unit testing allows to test your functionality in isolation. This usually means that instead of performing real work (e.g. sending HTTP request to LLM), it needs to be replaced with something that only simulates the work and return some predefined result, which will be used in testing class. As Semantic Kernel developers, we want to make sure that it’s easy to write unit tests for classes that use main library components – Kernel, Plugins and Functions.<\/span><\/p>\n

This article shows how to test the functionality in .NET applications which use Semantic Kernel. For demonstration purposes, we will use <\/span>xUnit<\/code> as test framework and Moq<\/code><\/span>\u00a0as mocking library.<\/span>\u00a0<\/span><\/p>\n

Let’s take a common scenario, where there is a service that does some work, and <\/span>Kernel<\/span> object is a dependency of that service:<\/span>\u00a0<\/span><\/p>\n

public class MyService(Kernel kernel) \r\n{ \r\n    private readonly Kernel _kernel = kernel; \r\n \r\n    public async Task<FunctionResult> DoWorkAsync(KernelFunction function) \r\n        => await _kernel.InvokeAsync(function); \r\n \r\n    public async Task<FunctionResult> DoWorkAsync(string pluginName, string functionName) \r\n        => await _kernel.InvokeAsync(pluginName, functionName); \r\n \r\n    public async Task<FunctionResult> DoWorkAsync(string prompt) \r\n        => await _kernel.InvokePromptAsync(prompt); \r\n}<\/code><\/pre>\n
In real-world applications, such services will probably contain more logic, but for simplicity, our service example will call available <\/span>Kernel<\/code> methods and return its result.<\/span>\u00a0<\/span><\/p>\n
It’s always possible to create a wrapper around Kernel<\/code> (<\/span>in similar way how \u00a0<\/span>MyService<\/code> looks now) and use it for mocking and testing purposes, but this article focuses on testing Kernel<\/code> <\/span>and functions directly.<\/span>\u00a0<\/span><\/p>\n
Kernel<\/code> is a container that holds services and plugins, which means that we don\u2019t want to mock <\/span>Kernel<\/span> directly. Instead, we want to mock its services and plugins. As presented in MyService<\/code><\/span>\u00a0class, Kernel<\/code> <\/span>has different ways how to execute a function and we will cover them in more detail. Next examples include:<\/span>\u00a0<\/span><\/p>\n
    \n
  1. Plugins \u2013 mocking calls to plugin functions.<\/span>\u00a0<\/span><\/li>\n
  2. Services \u2013 mocking calls to an LLM.<\/span>\u00a0<\/span><\/li>\n
  3. Streaming \u2013 additional considerations when using streaming with both plugin functions and LLM.<\/span><\/li>\n<\/ol>\n
    InvokeAsync with KernelFunction<\/span><\/b>\u00a0<\/span><\/p>\n
    When we do <\/span>kernel.InvokeAsync(function)<\/code>, <\/span>InvokeAsync<\/code> method performs <\/span>function.InvokeAsync(...)<\/code>, which means that we are interested in mocking\u00a0 <\/span>KernelFunction <\/code>behaviour.<\/span>\u00a0<\/span><\/p>\n
    At the moment, KernelFunction<\/code><\/span> can be either KernelFunctionFromMethod<\/code><\/span>, which is some method in C# code, or KernelFunctionFromPrompt<\/code><\/span>, which is constructed using prompt template and which will perform a request to LLM. The difference between these 2 types of KernelFunction<\/code> <\/span>defines how they need to be mocked.<\/span>\u00a0<\/span><\/p>\n
    In case when <\/span>Kernel<\/span> needs to invoke KernelFunctionFromMethod<\/code><\/span>, we can simply define a method in our test that will return desired result. For this purpose, we can use KernelFunctionFactory<\/code><\/span>, which is static helper class to create different types of functions. Method KernelFunctionFactory.CreateFromMethod<\/code><\/span>\u00a0accepts delegate, so we can define our function with desired result as lambda expression, local function or pass existing function as a parameter. For simplicity, we will use lambda expression.<\/span>\u00a0<\/span><\/p>\n
    Our first method in MyService<\/code><\/span>\u00a0accepts KernelFunction<\/code><\/span>, so we will create our test function and pass it to the service, where it will be invoked. Here is how the test will look like:<\/span>\u00a0<\/span><\/p>\n
    [Fact] \r\npublic async void DoWorkWithFunction() \r\n{ \r\n    \/\/ Arrange \r\n    var function = KernelFunctionFactory.CreateFromMethod(() => \"Function result value\"); \r\n    var kernel = new Kernel(); \r\n \r\n    var service = new MyService(kernel); \r\n \r\n    \/\/ Act \r\n    var result = await service.DoWorkAsync(function); \r\n \r\n    \/\/ Assert \r\n    Assert.Equal(\"Function result value\", result.ToString()); \r\n}<\/code><\/pre>\n
    InvokeAsync with Function and Plugin names<\/span><\/b>\u00a0<\/span><\/p>\n
    Kernel<\/span> holds a collection of plugins, so anytime we want to change <\/span>Kernel<\/span> behaviour in our tests, we need to pass it with registered plugins that don’t perform a real work and contain some mock behaviour instead.<\/span>\u00a0<\/span><\/p>\n
    Our second method in MyService<\/code> <\/span>doesn’t use KernelFunction<\/code><\/span>, it accepts Function and Plugin names. This means, that when we initialize our <\/span>Kernel<\/span>, we need to register some plugin with test behaviour, which will satisfy our test case.<\/span>\u00a0<\/span><\/p>\n
    To create test plugin, we can use static KernelPluginFactory<\/code> <\/span>class, which will help to create our plugin. This class contains methods like KernelPluginFactory.CreateFromType<T><\/code><\/span>\u00a0and KernelPluginFactory.CreateFromObject<\/code><\/span>, which are useful when our KernelFunction<\/code><\/span> is defined in separate class. These methods will use reflection to get a metadata about plugin class and create KernelPlugin<\/code><\/span>\u00a0instance.<\/span>\u00a0<\/span><\/p>\n
    For testing scenarios, there is a simpler way how to initialize plugin –KernelPluginFactory.CreateFromFunctions<\/code><\/span>\u00a0method. It accepts plugin name and enumeration of KernelFunction<\/code><\/span>\u00a0instances. Based on previous example, we already know how to create a function with mocked behaviour using KernelFunctionFactory<\/code><\/span>, which means that we can combine two approaches together, which will result in following test:<\/span>\u00a0<\/span><\/p>\n
    [Fact] \r\npublic async Task DoWorkWithFunctionName() \r\n{ \r\n    \/\/ Arrange \r\n    var function = KernelFunctionFactory.CreateFromMethod(() => \"Function result value\", \"MyFunction\"); \r\n    var plugin = KernelPluginFactory.CreateFromFunctions(\"MyPlugin\", [function]); \r\n    var plugins = new KernelPluginCollection([plugin]); \r\n \r\n    var kernel = new Kernel(plugins: plugins); \r\n \r\n    var service = new MyService(kernel); \r\n \r\n    \/\/ Act \r\n    var result = await service.DoWorkAsync(\"MyPlugin\", \"MyFunction\"); \r\n \r\n    \/\/ Assert \r\n    Assert.Equal(\"Function result value\", result.ToString()); \r\n} <\/code><\/pre>\n
    As soon as we create our KernelPlugin<\/code><\/span>, we can initialize <\/span>Kernel<\/span> and pass instance of KernelPluginCollection<\/code><\/span>, which will hold our test plugin with mocked behaviour. Then, we inject kernel instance to the service, and it will use test function to return configured result.<\/span>\u00a0<\/span><\/p>\n
    InvokePromptAsync<\/span><\/b>\u00a0<\/span><\/p>\n
    In previous examples, we mocked KernelFunctionFromMethod<\/code><\/span>\u00a0behaviour, but <\/span>Kernel<\/span> can also hold KernelFunctionFromPrompt<\/code><\/span> functions. As already described, these functions will perform requests to LLM, which means that most probably HttpClient<\/code> <\/span>will be involved and in this case more mocking logic will be required.<\/span>\u00a0<\/span><\/p>\n
    When we do kernel.InvokePromptAsync(...)<\/code><\/span>, we just call an extension method that will create instance of KernelFunctionFromPrompt<\/code> <\/span>and immediately invoke it.<\/span>\u00a0<\/span><\/p>\n
    KernelFunctionFromPrompt<\/code><\/span> during invocation will render prompt using configured prompt template and pass that prompt to service, which will be responsible to send a request to LLM. KernelFunctionFromPrompt<\/code><\/span> will try to find an instance of IChatCompletionService<\/code><\/span> or ITextGenerationService<\/code><\/span> service to perform the request. Which means that we can mock IChatCompletionService<\/code> <\/span>and return desired result. We also need to register IChatCompletionService<\/code><\/span>\u00a0in <\/span>Kernel<\/span>, because that’s the place where our function will look for it.<\/span>\u00a0<\/span><\/p>\n
    To define test behaviour of IChatCompletionService<\/code><\/span>, it’s possible to implement that interface with fake class, but easier way would be to use mocking library of your choice. As soon as we have mocked behaviour in place, we need to register it in <\/span>Kernel<\/span> and invoke our prompt function. Here is an example how the test could look like:<\/span>\u00a0<\/span><\/p>\n
    [Fact] \r\npublic async Task DoWorkWithPrompt() \r\n{ \r\n    \/\/ Arrange \r\n    var mockChatCompletion = new Mock<IChatCompletionService>(); \r\n    mockChatCompletion \r\n        .Setup(x => x.GetChatMessageContentsAsync( \r\n            It.IsAny<ChatHistory>(), \r\n            It.IsAny<PromptExecutionSettings>(), \r\n            It.IsAny<Kernel>(), \r\n            It.IsAny<CancellationToken>())) \r\n        .ReturnsAsync([new ChatMessageContent(AuthorRole.Assistant, \"AI response\")]); \r\n \r\n    var kernelBuilder = Kernel.CreateBuilder(); \r\n    kernelBuilder.Services.AddSingleton(mockChatCompletion.Object); \r\n \r\n    var kernel = kernelBuilder.Build(); \r\n    var service = new MyService(kernel); \r\n \r\n    \/\/ Act \r\n    var result = await service.DoWorkAsync(\"Prompt to AI\"); \r\n \r\n    \/\/ Assert \r\n    Assert.Equal(\"AI response\", result.ToString()); \r\n} <\/code><\/pre>\n
    As alternative, if AI connector accepts HttpClient<\/code> <\/span>as parameter, it’s also possible to define test behaviour and mock HttpClient<\/code><\/span>.<\/span>\u00a0<\/span><\/p>\n
    Testing streaming scenarios<\/span><\/b>\u00a0<\/span><\/p>\n
    In case of streaming, the tests will look very similar to previous examples, with small differences in places where we define our test response and mock IChatCompletionService<\/code><\/span>.<\/span>\u00a0<\/span><\/p>\n
    Let’s rewrite MyService<\/code><\/span>\u00a0to call <\/span>Kernel<\/span> streaming methods:<\/span>\u00a0<\/span><\/p>\n
    public class MyService(Kernel kernel) \r\n{ \r\n    private readonly Kernel _kernel = kernel; \r\n \r\n    public IAsyncEnumerable<T> DoWorkStreaming<T>(KernelFunction function) \r\n        => _kernel.InvokeStreamingAsync<T>(function); \r\n \r\n    public IAsyncEnumerable<T> DoWorkStreaming<T>(string pluginName, string functionName) \r\n        => _kernel.InvokeStreamingAsync<T>(pluginName, functionName); \r\n \r\n    public IAsyncEnumerable<T> DoWorkStreaming<T>(string prompt) \r\n        => _kernel.InvokePromptStreamingAsync<T>(prompt); \r\n} <\/code><\/pre>\n
    Based on this service, the tests will look like following:<\/span><\/span>\u00a0<\/span><\/p>\n
    [Fact] \r\npublic async Task DoWorkStreamingWithFunction() \r\n{ \r\n    \/\/ Arrange \r\n    async IAsyncEnumerable<int> TestMethod() \r\n    { \r\n        yield return 1; \r\n        yield return 2; \r\n        yield return 3; \r\n    } \r\n \r\n    var function = KernelFunctionFactory.CreateFromMethod(TestMethod); \r\n \r\n    var kernel = new Kernel(); \r\n \r\n    var service = new MyService(kernel); \r\n \r\n    \/\/ Act \r\n    var result = new List<int>(); \r\n    await foreach (var item in service.DoWorkStreaming<int>(function)) \r\n    { \r\n        result.Add(item); \r\n    } \r\n \r\n    \/\/ Assert \r\n    Assert.Equal([1, 2, 3], result); \r\n} \r\n \r\n[Fact] \r\npublic async Task DoWorkWithFunctionName() \r\n{ \r\n    \/\/ Arrange \r\n    async IAsyncEnumerable<int> TestMethod() \r\n    { \r\n        yield return 1; \r\n        yield return 2; \r\n        yield return 3; \r\n    } \r\n \r\n    var function = KernelFunctionFactory.CreateFromMethod(TestMethod); \r\n    var plugin = KernelPluginFactory.CreateFromFunctions(\"MyPlugin\", [function]); \r\n    var plugins = new KernelPluginCollection([plugin]); \r\n \r\n    var kernel = new Kernel(plugins: plugins); \r\n \r\n    var service = new MyService(kernel); \r\n \r\n    \/\/ Act \r\n    var result = new List<int>(); \r\n    await foreach (var item in service.DoWorkStreaming<int>(function)) \r\n    { \r\n        result.Add(item); \r\n    } \r\n \r\n    \/\/ Assert \r\n    Assert.Equal([1, 2, 3], result); \r\n} \r\n \r\n[Fact] \r\npublic async Task DoWorkWithPrompt() \r\n{ \r\n    \/\/ Arrange \r\n    var mockChatCompletion = new Mock<IChatCompletionService>(); \r\n    mockChatCompletion \r\n        .Setup(x => x.GetStreamingChatMessageContentsAsync( \r\n            It.IsAny<ChatHistory>(), \r\n            It.IsAny<PromptExecutionSettings>(), \r\n            It.IsAny<Kernel>(), \r\n            It.IsAny<CancellationToken>())) \r\n        .Returns(new List<StreamingChatMessageContent>() { new(AuthorRole.Assistant, \"AI response\") }.ToAsyncEnumerable()); \r\n \r\n    var kernelBuilder = Kernel.CreateBuilder(); \r\n    kernelBuilder.Services.AddSingleton(mockChatCompletion.Object); \r\n \r\n    var kernel = kernelBuilder.Build(); \r\n    var service = new MyService(kernel); \r\n \r\n    \/\/ Act \r\n    var result = new List<string>(); \r\n    await foreach (var item in service.DoWorkStreaming<string>(\"Prompt to AI\")) \r\n    { \r\n        result.Add(item); \r\n    } \r\n \r\n    \/\/ Assert \r\n    Assert.Equal([\"AI response\"], result); \r\n}<\/code><\/pre>\n
    In case of streaming, we still use KernelFunctionFactory<\/code> <\/span>and KernelPluginFactory<\/code> <\/span>as helper classes to initialize test plugins and functions.<\/span>\u00a0<\/span><\/p>\n
    The only difference is the result of mocked functions – instead of returning value directly, we return IAsyncEnumerable<T><\/code><\/span>. We also mock IChatCompletionService<\/code> <\/span>differently – instead of mocking GetChatMessageContentsAsync<\/code> <\/span>method, we mock GetStreamingChatMessageContentsAsync<\/code><\/span>. We also use ToAsyncEnumerable<\/code><\/span>\u00a0method from System.Linq.Async<\/code><\/span>\u00a0package to convert our result list to IAsyncEnumerable<\/code> <\/span>in easier way.<\/span>\u00a0<\/span><\/p>\n
    Summary<\/span><\/b>\u00a0<\/span><\/p>\n
    Provided examples show how to write unit tests for services which use <\/span>Kernel<\/span> as a dependency. Although these examples focus on unit testing, similar approaches can be used for integration testing.<\/span>\u00a0<\/span><\/p>\n
    We\u2019re always interested in hearing from you. If you have feedback, questions or want to discuss further, feel free to reach out to us and the community on the\u00a0discussion boards<\/a> on GitHub! We would also love your support, if you’ve enjoyed using Semantic Kernel, give us a star on GitHub<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Hi all, Testing is an integral part of the software development process. Unit testing allows to test your functionality in isolation. This usually means that instead of performing real work (e.g. sending HTTP request to LLM), it needs to be replaced with something that only simulates the work and return some predefined result, which will […]<\/p>\n","protected":false},"author":149071,"featured_media":2365,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[17,1],"tags":[48,9,68],"class_list":["post-2523","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-announcements","category-semantic-kernel","tag-ai","tag-semantic-kernel","tag-unit-testing"],"acf":[],"blog_post_summary":"
    Hi all, Testing is an integral part of the software development process. Unit testing allows to test your functionality in isolation. This usually means that instead of performing real work (e.g. sending HTTP request to LLM), it needs to be replaced with something that only simulates the work and return some predefined result, which will […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/2523","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/users\/149071"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/comments?post=2523"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/2523\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media\/2365"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media?parent=2523"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/categories?post=2523"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/tags?post=2523"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}