{"id":3477,"date":"2024-10-03T14:19:12","date_gmt":"2024-10-03T21:19:12","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/semantic-kernel\/?p=3477"},"modified":"2025-02-10T10:18:28","modified_gmt":"2025-02-10T18:18:28","slug":"microsoft-hackathon-project-micronaire-using-semantic-kernel","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/agent-framework\/microsoft-hackathon-project-micronaire-using-semantic-kernel\/","title":{"rendered":"Microsoft Hackathon: Project Micronaire using Semantic Kernel"},"content":{"rendered":"

During our internal Microsoft Hackathon, there were a number of projects using Semantic Kernel. Today we’d like to highlight one of the projects focused on using Semantic Kernel and the new Vector Store abstraction. I’m going to turn it over to the Hackathon team to share more about their project and work using Semantic Kernel.<\/p>\n

Summary<\/h4>\n

In Microsoft’s 2024 Global Hackathon, we built the prototype for Micronaire as well as a demo project that showcased the capabilities of the library. We built a Romeo and Juliet chat bot that used RAG (Retrieval Augmented Generation) to recall paragraphs of the original Romeo and Juliet text. We utilized the new Vector Store abstraction from Semantic Kernel with their implementation for the Qdrant Database in order to store embeddings of these paragraphs. Once we had a working chat bot, we used Micronaire to evaluate its performance and generate graphs of the data.<\/p>\n

RAG pipelines enable developers to augment their chat experiences with informational documents that their agents can leverage to provide better answers. Evaluating these pipelines is a new area of study, with solutions like\u00a0RAGChecker<\/a>\u00a0and\u00a0RAGAS<\/a>\u00a0being state of the art frameworks that are implemented in Python. This project aims to take these ideas and bring them to DotNet through\u00a0Semantic Kernel<\/a>.<\/p>\n

Micronaire brings actionable metrics to RAG pipeline evaluation by taking a set of ground truth questions and answers as well as a RAG pipeline, evaluating the pipeline against the ground truth using our metrics (see below), and then producing an evaluation report.<\/p>\n

Micronaire has been released as open-source code here: microsoft\/micronaire: A RAG evaluation pipeline for Semantic Kernel (github.com)<\/a><\/p>\n

Here’s a link to the Hackathon Project video to learn more: https:\/\/learn-video.azurefd.net\/vod\/player?id=60a35fb0-3c05-454a-9307-afec267c7854<\/a><\/p>\n

Why the name?<\/h4>\n

Micronaire<\/a>\u00a0is a measure of the air permeability of cotton fiber and is an indication of fineness and maturity. This plays into the evaluation of RAG (like rag textiles).<\/p>\n

Semantic Kernel Usage<\/h4>\n

Micronaire uses Semantic Kernel in two ways. First, it uses Semantic Kernel as a way to target RAG pipelines that users of the library have written in Semantic Kernel\u2019s C-Sharp implementation. Second, it uses Semantic Kernel as an AI Orchestrator to run its evaluation and call text completion APIs.<\/p>\n

Using Micronaire<\/h4>\n

After bringing in the Micronaire library, users need to implement Micronaire\u2019s expected RAG interface for their Semantic Kernel chat bot experience. The interface is shown below:<\/p>\n

\"Image<\/a><\/p>\n

Next, users must generate a set of ground truth questions and answers for their evaluation in a JSON list. A simple example of this is shown below:<\/p>\n

\"Image<\/a><\/p>\n

Users must register Micronaire with dependency injection to allow for Micronaire objects to be created:<\/p>\n

\"Image<\/a><\/p>\n

Finally, users can grab the Micronaire evaluator from the container and call the evaluation function, passing in their RAG pipeline as well as the path to their ground truth data:<\/p>\n

\"Image<\/a><\/p>\n

Metrics<\/h4>\n

Micronaire will generate a set of metrics across different categories. These categories represent all the approaches implemented in the library. Most of these approaches use claim evaluation (see below) to pull out fine-grained claims made by<\/p>\n

LLM Evaluation<\/h4>\n

LLM evaluation uses an LLM to try to evaluate the performance of the RAG pipeline. It uses direct LLM calls which depends on the performance of the evaluating LLM, so using the strongest model possible is best. The following are the metrics that are generated, which are inspired by work done here: Evaluation and monitoring metrics for generative AI – Azure AI Studio | Microsoft Learn<\/a>.<\/p>\n

Groundedness<\/h4>\n

Measures how well the model’s generated answers align with information from the source data (user-defined context). Higher groundedness is better.<\/p>\n

Relevance<\/h4>\n

Relevance measures the extent to which the model’s generated responses are pertinent and directly related to the queries given. Higher relevance is better.<\/p>\n

Coherence<\/h4>\n

Measures how well the language model can produce output that flows smoothly, reads naturally, and resembles human-like language. A higher coherence is better.<\/p>\n

Fluency<\/h4>\n

Measures the grammatical proficiency of a generative AI’s predicted answer. Higher fluency is better.<\/p>\n

Retrieval Score<\/h4>\n

Measures the extent to which the model’s retrieved documents are pertinent and directly related to the queries given. Higher retrieval scores are better.<\/p>\n

Similarity<\/h4>\n

Measures the similarity between a source data (ground truth) sentence and the generated response by an AI model. A higher similarity is better.<\/p>\n

Overall Claim Evaluation<\/h3>\n

Overall claim evaluation uses extracted claims to evaluate the overall performance of the RAG pipeline. It uses the following metrics:<\/p>\n

Precision<\/h4>\n

Precision tracks the proportion of correct claims to the total number of claims from the generated answer.<\/p>\n

Recall<\/h4>\n

Recall tracks the proportion of correct claims to the total number of ground truth claims.<\/p>\n

F1 Score<\/h4>\n

The F1 Score is the harmonic mean of the precision and recall scores. It can be used as an overall performance metric for the RAG pipeline.<\/p>\n

Retrieval Claim Evaluation<\/h3>\n

Retrieval claim evaluation uses extracted claims to evaluate just the retrieval (the R in RAG) part of the RAG pipeline. It uses the following metrics:<\/p>\n

Claim Recall<\/h4>\n

This is the proportion of claims in the ground-truth answer that are entailed in the retrieved context by the retriever. We say a ground-truth answer claim is covered if it is entailed in the retrieved context claims.<\/p>\n

\"Image<\/a><\/h3>\n

Context Precision<\/h4>\n

This is the proportion of claims in the retrieved context that are entailed in the ground-truth answer. A context claim is relevant if it is entailed in the ground-truth answer claims.<\/p>\n

\"Image<\/a><\/p>\n

<\/a>Generation Claim Evaluation<\/h3>\n

Generation claim evaluation uses extracted claims to evaluate just the generation (the G in RAG) part of the RAG pipeline. It uses the following metrics defined in \u00a0RAGChecker<\/a>:<\/p>\n

Faithfulness<\/h4>\n

This is a measure of how faithful generator uses the retrieved context to generate final response. A higher faithfulness means a better pipeline since it means generator is actually using the retrieved context.<\/p>\n

\"Image<\/a><\/p>\n

Relevant Noise Sensitivity<\/h4>\n

This is a measure of how sensitive the generator is to noise mixed with useful information in a relevant retrieved context. \u00a0A lower noise sensitivity is better.<\/p>\n

\"Image<\/a><\/p>\n

Irrelevant Noise Sensitivity<\/h4>\n

This is a measure of how sensitive the generator is to noise in a retrieved context that is irrelevant. A lower noise sensitivity is better<\/p>\n

\"Image<\/a><\/p>\n

Hallucination<\/h4>\n

This is a measure of the propensity of the generator to make up incorrect claims with no backing by retrieved context. A lower hallucination rate is better.<\/p>\n

\"Image<\/a><\/p>\n

Self-Knowledge Score<\/h4>\n

This is a measure of the generator\u2019s ability to make up correct claims that are not backed by the retrieved context. A higher Self-Knowledge score is better.<\/p>\n

\"Image<\/a><\/p>\n

Context Utilization<\/h4>\n

This is a measure of how effectively the generator uses retrieved context to generate the ground truth answer.<\/p>\n

\"Image<\/a><\/h4>\n

Claim Extraction<\/span>\u00a0<\/span><\/h4>\n

Claim extraction is the process of extracting a set of fine-grained claims of (potentially false) statements from a text. Micronaire achieves this through a chain-of-thought prompt that attempts to first extract subject, relationship, object triples and then combine those into a simple sentence that represents a single claim in a text.<\/span>\u00a0<\/span><\/p>\n

Hallucination =Number of incorrect claims with no backing by retrieved context \/ Total number of claims in the generated response<\/p>\n

Conclusion<\/strong><\/h4>\n

We want to thank the Project Micronaire hackathon team for joining the blog today and sharing their work.\u00a0Please reach out if you have any questions or feedback through our\u00a0Semantic Kernel GitHub Discussion Channel<\/a>. We look forward to hearing from you!\u00a0We would also love your support, if you\u2019ve enjoyed using Semantic Kernel, give us a star on\u00a0GitHub<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

During our internal Microsoft Hackathon, there were a number of projects using Semantic Kernel. Today we’d like to highlight one of the projects focused on using Semantic Kernel and the new Vector Store abstraction. I’m going to turn it over to the Hackathon team to share more about their project and work using Semantic Kernel. […]<\/p>\n","protected":false},"author":149071,"featured_media":2302,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[48,63,9],"class_list":["post-3477","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-semantic-kernel","tag-ai","tag-microsoft-semantic-kernel","tag-semantic-kernel"],"acf":[],"blog_post_summary":"

During our internal Microsoft Hackathon, there were a number of projects using Semantic Kernel. Today we’d like to highlight one of the projects focused on using Semantic Kernel and the new Vector Store abstraction. I’m going to turn it over to the Hackathon team to share more about their project and work using Semantic Kernel. […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/3477","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/users\/149071"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/comments?post=3477"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/3477\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media\/2302"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media?parent=3477"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/categories?post=3477"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/tags?post=3477"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}