Unlock the Power of Telemetry in Semantic Kernel SDK

Nilesh Acharya

August 3rd, 20230 3

As more and more customers are becoming reliant on the Semantic Kernel SDK to create sophisticated and powerful applications, we have heard the need for greater transparency into end-to-end traceability and debugging. For example, if you want to know how a specific plugin can impact performance or looking for a way to debug a request this blog post is for you.

We have chosen to use Azure Application Insights, but feel free to use any telemetry provider of your choice, including examples such as OpenTelemetry or Prometheus. Apologies for the longer than usual blog post but we have some goodness to share!

Key Benefits of effective product telemetry

So why you should use effective product telemetry? Product telemetry refers to the process of collecting and analyzing data from software applications to gain insights into the following:

Bug Detection and Debugging: Telemetry enables near real-time monitoring of applications, allowing prompt detection and diagnosis of issues or bugs. This leads to faster resolution times and improves overall software quality. To do this successfully your application needs to have tracing.
Performance Optimization: By tracking application performance metrics, you can identify bottlenecks and areas for optimization. This data-driven approach ensures that the application operates at its best, enhancing user experience.
Proactive Maintenance: Telemetry allows proactive maintenance as you can anticipate potential problems and address them before they escalate, reducing downtime and enhancing product reliability.
Data-Driven Decision Making: With telemetry, decisions can be based on data rather than assumptions, resulting in more effective and successful product development strategies.

Get Started with Telemetry in Semantic Kernel SDK

Logging is implemented with ILogger interface from Microsoft.Extensions.Logging namespace, Semantic Kernel uses the following log levels:

Information – This is the default enabled log level. This level provides information about the general flow of the application and contains the following data:
- AI model used to create a plan.
- Plan creation status (Success/Failed).
- Plan creation execution time (in milliseconds).
- Created plan without plugin arguments.
- AI model used to execute a plugin.
- Plugin execution status (Success/Failed)
- Plugin execution time (in milliseconds)
Debug – contains more detailed messages without sensitive data. This can be enabled in production environments.
Warning – includes information about unusual events that don’t cause the application to fail.
Error – used for logging exception details.
Trace – this log level is the most verbose and should not be enabled in production environments since it may contain sensitive data. This is a great choice for test environments for increased levels of observability. Logged information includes:
- Goal/Ask to create a plan.
- Prompt (template and rendered version) for AI to create a plan.
- Created plan with function arguments (arguments may contain sensitive data).
- Prompt (template and rendered version) for AI to execute a plugin.

Below snippet illustrates how to enable logging for Kernel instance:

var kernel = new KernelBuilder().WithLogger(logger);

To enable logging for Planner instance (metering and tracing will be enabled as well):

var planner = new SequentialPlanner(kernel, plannerConfig).WithInstrumentation(logger);

Tracing

To apply tracing to aid with data flow observability you can use the Activity Class from System.Diagnostics namespace. The available activity sources are the list below:

Microsoft.SemanticKernel.Planning.Action.InstrumentedActionPlanner – creates activities for ActionPlanner.
Microsoft.SemanticKernel.Planning.Sequential.InstrumentedSequentialPlanner – creates activities for SequentialPlanner.
Microsoft.SemanticKernel.Planning.Stepwise.StepwisePlanner – creates activities for StepwisePlanner.
Microsoft.SemanticKernel.Planning.Plan – creates activities for Plan.
Microsoft.SemanticKernel.SkillDefinition.SKFunction – creates activities for SKFunction.

To subscribe to the available activity source use ‘Activity Listener’

var activityListener = new ActivityListener();

activityListener.ShouldListenTo =
    activitySource => activitySource.Name.StartsWith("Microsoft.SemanticKernel", StringComparison.Ordinal);

ActivitySource.AddActivityListener(activityListener);

To subscribe to all activities sources in Semantic Kernel you can use the following:

activitySource.Name.StartsWith("Microsoft.SemanticKernel", StringComparison.Ordinal)

If you want to subscribe to specific activity source, the following example shows how to subscribe based a condition to subscribe to SKFunction only.

activitySource.Name.Equals("Microsoft.SemanticKernel.SkillDefinition.SKFunction", StringComparison.Ordinal)

Once applied you will be able to view traces like the below from Azure Application Insights.

Or integrate with log queries to view end to end traces across key components including plugins, planners and model end points.

Metering

This is great when you require understanding of your applications performance. Metering is implemented with Meter class from System.Diagnostics.Metrics namespace. The following meters are available:

SemanticKernel.Planning.Action.InstrumentedActionPlanner – captures metrics for ActionPlanner. List of metrics:
- ActionPlanner.CreatePlan.ExecutionTime- execution time of plan creation (in milliseconds)
SemanticKernel.Planning.Sequential.InstrumentedSequentialPlanner – captures metrics for SequentialPlanner. List of metrics:
- SequentialPlanner.CreatePlan.ExecutionTime- execution time of plan creation (in milliseconds)
SemanticKernel.Planning.Stepwise.StepwisePlanner – captures metrics for StepwisePlanner. List of metrics:
- StepwisePlanner.CreatePlan.ExecutionTime- execution time of plan creation (in milliseconds)
SemanticKernel.Planning.Plan – captures metrics for Plan. List of metrics:
- Plan.Execution.ExecutionTime- plan execution time (in milliseconds)
- Plan.Execution.ExecutionTotal- total number of plan executions
- Plan.Execution.ExecutionSuccess- number of successful plan executions
- Plan.Execution.ExecutionFailure- number of failed plan executions
SemanticKernel.SkillDefinition.SKFunction – captures metrics for SKFunction. List of metrics:
- <SkillName><FunctionName>.ExecutionTime- function execution time (in milliseconds)
- <SkillName><FunctionName>.ExecutionTotal- total number of function executions
- <SkillName><FunctionName>.ExecutionSuccess- number of successful function executions
- <SkillName><FunctionName>.ExecutionFailure- number of failed function executions

Depending on your telemetry monitoring choice, subscribing to events may vary. The following example shows you can subscribe to meters and export metrics using Application Insights using MeterListener:

var meterListener = new MeterListener();

meterListener.InstrumentPublished = (instrument, listener) =>
{
    if (instrument.Meter.Name.StartsWith("Microsoft.SemanticKernel", StringComparison.Ordinal))
    {
        listener.EnableMeasurementEvents(instrument);
    }
};

// Set callback to specific numeric type - double.
meterListener.SetMeasurementEventCallback<double>((instrument, measurement, tags, state) =>
{
    // Export to Application Insights using telemetry client instance
    telemetryClient.GetMetric(instrument.Name).TrackValue(measurement);
});

meterListener.Start();

It’s possible to control for what meters to subscribe. For example, following condition will allow to subscribe to all meters in Semantic Kernel:

instrument.Meter.Name.StartsWith("Microsoft.SemanticKernel", StringComparison.Ordinal)

If you want to subscribe to a specific meter, you can do that too. The following example shows a condition to subscribe to meter for SKFunction only

instrument.Meter.Name.Equals("Microsoft.SemanticKernel.SkillDefinition.SKFunction", StringComparison.Ordinal)

Once you have implemented metering you can build metric views similar to the one below from Azure Application Insights to observe how different plugin execution times compare.

In closing, telemetry forms the backbone of modern software development, and the Semantic Kernel SDK. By harnessing telemetry data, you can gain invaluable insights, improve application performance, and build products that truly resonate with users. Embrace telemetry in your Semantic Kernel SDK projects, and unlock the potential to elevate your applications to new heights of success. Happy coding!

Next Steps: