Quick Introduction to Microsoft.Extensions.VectorData<\/h2>\n
Microsoft.Extensions.VectorData<\/strong> is a set of code .NET libraries designed for managing vector-based data in .NET applications. These libraries provide a unified layer of C# abstractions for interacting with vector stores, enabling developers to handle embeddings and perform vector similarity queries efficiently.<\/p>\n To get a detailed overview of the library\u2019s architecture and capabilities, I recommend reading Luis\u2019s excellent blog post<\/a>.<\/p>\n In this blog post, we\u2019ll showcase two real-world use cases:<\/p>\n Using Qdrant<\/strong> locally for semantic search.<\/p>\n<\/li>\n Leveraging Azure AI Search<\/strong> for enterprise-scale vector search.<\/p>\n<\/li>\n<\/ol>\n To run the demos, you need to use one of the models provided by Ollama for the embeddings generations. In this sample, the model used is all-minilm.<\/p>\n Install Ollama<\/a>.<\/p>\n<\/li>\n Download the all-minilm model<\/a>.<\/p>\n<\/li>\n An OCI compliant container runtime, such as:<\/p>\n Qdrant<\/strong> is a vector similarity search engine that provides a production-ready service with a convenient API to store, search, and manage points (i.e. vectors) with an additional payload. It\u2019s perfect for applications that require efficient similarity searches. You can easily run Qdrant locally in a Docker container, making it a developer-friendly choice.<\/p>\n For setup instructions, refer to the Qdrant Quickstart Guide<\/a>. And, as for reference, this is a sample command to run a local container instance:<\/p>\n Once the container is created, you can check it in Docker.<\/p>\n Semantic Kernel<\/strong> provides a built-in connector for Qdrant, enabling .NET developers to store embeddings and execute vector-based queries seamlessly. This connector is built on top of This integration combines Qdrant\u2019s high performance with Semantic Kernel\u2019s ease of use.<\/p>\n To learn more about the connector, visit the official documentation for Semantic Kernel Vector Store Qdrant connector<\/a>.<\/p>\n Let’s see a sample class that implements and runs this demo.<\/p>\n Once the demo is run, this is the sample output:<\/p>\n Using Qdrant for semantic search offers the advantage of scalable, high-speed similarity search<\/strong>, making it an excellent choice for applications requiring large-scale vector data management. Additionally, you have the option to run Qdrant Cloud on Microsoft Azure<\/a>.<\/p>\n Azure AI Search<\/strong> is Microsoft\u2019s search-as-a-service offering. It integrates traditional search capabilities with AI-powered features like semantic and vector search. Built for scalability and reliability, it is an ideal solution for enterprise applications requiring advanced search functionality<\/strong>. You can learn more about Azure AI Search<\/a>.<\/p>\n For this sample, we will use the integrated vectorization in Azure AI Search<\/a>, which improves indexing and querying by converting documents and queries into vectors. <\/p>\n This connector is built on top of For more information, refer to the Azure AI Search connector documentation<\/a>.<\/p>\n Here is a console command sample on how to add the User Secrets:<\/p>\n Let’s see a sample class that implements and runs this demo.<\/p>\n Once the demo is run, this is the sample output:<\/p>\n And we can see the new index with the Movie fields in the Azure Portal in the Azure AI Search service.<\/p>\n Azure AI Search<\/a> provides enterprise-grade scalability and integration<\/strong>, making it a robust solution for production-ready applications requiring advanced semantic search. Additionally, AI Search includes built-in security features<\/strong>, such as encryption and secure authentication, to protect your data. It also adheres to compliance standards, ensuring that your search solutions meet regulatory requirements.<\/p>\n Each semantic search demo is implemented as a .NET 9 Console Application<\/strong>. The codebase for the samples can be traced back to the original demo provided by Luis<\/a>, with extensions for both Azure AI Search and Qdrant scenarios.<\/p>\n A shared class represents a Movie<\/strong> entity, which includes:<\/p>\n You can browse the github repository with the complete code samples<\/a>.<\/p>\n The journey with Microsoft.Extensions.VectorData<\/strong> doesn\u2019t stop here. You can choose other connectors like SQLite in memory<\/strong>, Pinecone<\/strong> or Redis<\/strong>; enabling developers to run lightweight semantic search solutions locally. This feature will be perfect for scenarios where performance and simplicity are essential.<\/p>\n And we are also working with partners like Elasticsearch are already building on top of Microsoft.Extensions.VectorData. You can learn more about this use case on Customer Case Study: Announcing the Microsoft Semantic Kernel Elasticsearch Connector<\/a>.<\/p>\n The combination of Microsoft.Extensions.VectorData<\/strong> and Semantic Kernel<\/strong>, allows .NET developers to build intelligent, scalable, and context-aware applications. Whether you\u2019re working on a small-scale project or a large enterprise system, these tools provide the foundation for delivering cutting-edge semantic search experiences.<\/p>\n Stay tuned for more tutorials and resources, and feel free to connect with us on social media for questions or feedback.\nHappy Coding!<\/p>\n","protected":false},"excerpt":{"rendered":" Discover how to use Microsoft.Extensions.VectorData to implement semantic search using Qdrant and Azure AI Search<\/p>\n","protected":false},"author":120281,"featured_media":54902,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[685,7781,756],"tags":[7797,568,7891,7885,7806],"class_list":["post-54901","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dotnet","category-ai","category-csharp","tag-dotnet-9","tag-ai","tag-azure-ai-search","tag-microsoft-extensions-ai","tag-vectordb"],"acf":[],"blog_post_summary":" Discover how to use Microsoft.Extensions.VectorData to implement semantic search using Qdrant and Azure AI Search<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/54901","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/users\/120281"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/comments?post=54901"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/54901\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media\/54902"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media?parent=54901"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/categories?post=54901"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/tags?post=54901"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
\n
\n
Semantic Search with Qdrant<\/h2>\n
What is Qdrant?<\/h3>\n
docker run -p 6333:6333 -p 6334:6334 -v $(pwd)\/qdrant_storage:\/qdrant\/storage:z qdrant\/qdrant<\/code><\/pre>\n![\"qdrant](\"https:\/\/devblogs.microsoft.com\/dotnet\/wp-content\/uploads\/sites\/10\/2024\/12\/05Qdrantrunningindocker.png\")
<\/p>\nQdrant and Semantic Kernel<\/h3>\n
Microsoft.Extensions.VectorData<\/code> and the official .NET Qdrant Client<\/a>.<\/p>\nScenario Overview – Qdrant<\/h3>\n
\n
\n
using Microsoft.Extensions.AI;\nusing Microsoft.Extensions.VectorData;\nusing Microsoft.SemanticKernel.Connectors.Qdrant;\nusing Qdrant.Client;\n\nvar vectorStore = new QdrantVectorStore(new QdrantClient(\"localhost\"));\n\n\/\/ get movie list\nvar movies = vectorStore.GetCollection<ulong, MovieVector<ulong>>(\"movies\");\nawait movies.CreateCollectionIfNotExistsAsync();\nvar movieData = MovieFactory<ulong>.GetMovieVectorList();\n\n\/\/ get embeddings generator and generate embeddings for movies\nIEmbeddingGenerator<string, Embedding<float>> generator =\n new OllamaEmbeddingGenerator(new Uri(\"http:\/\/localhost:11434\/\"), \"all-minilm\");\nforeach (var movie in movieData)\n{\n movie.Vector = await generator.GenerateEmbeddingVectorAsync(movie.Description);\n await movies.UpsertAsync(movie);\n}\n\n\/\/ perform the search\nvar query = \"A family friendly movie that includes ogres and dragons\";\nvar queryEmbedding = await generator.GenerateEmbeddingVectorAsync(query);\n\nvar searchOptions = new VectorSearchOptions()\n{\n Top = 2,\n VectorPropertyName = \"Vector\"\n};\n\nvar results = await movies.VectorizedSearchAsync(queryEmbedding, searchOptions);\nawait foreach (var result in results.Results)\n{\n Console.WriteLine($\"Title: {result.Record.Title}\");\n Console.WriteLine($\"Description: {result.Record.Description}\");\n Console.WriteLine($\"Score: {result.Score}\");\n Console.WriteLine();\n}<\/code><\/pre>\nTitle: Shrek\nDescription: Shrek is an animated film that tells the story of an ogre named Shrek who embarks on a quest to rescue Princess Fiona from a dragon and bring her back to the kingdom of Duloc.\nScore: 0.5013245344161987\n\nTitle: Lion King\nDescription: The Lion King is a classic Disney animated film that tells the story of a young lion named Simba who embarks on a journey to reclaim his throne as the king of the Pride Lands after the tragic death of his father.\nScore: 0.3225690722465515<\/code><\/pre>\nWhy Qdrant?<\/h3>\n
Semantic Search with Azure AI Search<\/h2>\n
What is Azure AI Search?<\/h3>\n
Azure AI Search and Semantic Kernel<\/h3>\n
Microsoft.Extensions.VectorData<\/code> and the official Azure AI Search libraries for .NET<\/a>.<\/p>\nScenario Overview – Azure AI Search<\/h3>\n
\n
\n
dotnet user-secrets init\ndotnet user-secrets set \"AZURE_AISEARCH_URI\" \"https:\/\/<AI Search Name>.search.windows.net\"\ndotnet user-secrets set \"AZURE_AISEARCH_SECRET\" \"AI Search Secret\"<\/code><\/pre>\nusing Microsoft.Extensions.AI;\nusing Microsoft.Extensions.VectorData;\nusing Azure;\nusing Azure.Search.Documents.Indexes;\nusing Microsoft.SemanticKernel.Connectors.AzureAISearch;\nusing Microsoft.Extensions.Configuration;\nusing Azure.Identity;\nusing Azure.Core;\n\n\/\/ get the search index client using Azure Default Credentials or Azure Key Credential with the service secret\nvar client = GetSearchIndexClient();\nvar vectorStore = new AzureAISearchVectorStore(searchIndexClient: client);\n\n\/\/ get movie list\nvar movies = vectorStore.GetCollection<string, MovieVector<string>>(\"movies\");\nawait movies.CreateCollectionIfNotExistsAsync();\nvar movieData = MovieFactory<string>.GetMovieVectorList();\n\n\/\/ get embeddings generator and generate embeddings for movies\nIEmbeddingGenerator<string, Embedding<float>> generator =\n new OllamaEmbeddingGenerator(new Uri(\"http:\/\/localhost:11434\/\"), \"all-minilm\");\nforeach (var movie in movieData)\n{\n movie.Vector = await generator.GenerateEmbeddingVectorAsync(movie.Description);\n await movies.UpsertAsync(movie);\n}\n\n\/\/ perform the search\nvar query = \"A family friendly movie that includes ogres and dragons\";\nvar queryEmbedding = await generator.GenerateEmbeddingVectorAsync(query);\n\n\/\/ show the results...<\/code><\/pre>\nTitle: Shrek\nDescription: Shrek is an animated film that tells the story of an ogre named Shrek who embarks on a quest to rescue Princess Fiona from a dragon and bring her back to the kingdom of Duloc.\nScore: 0.6672559<\/code><\/pre>\n![\"index](\"https:\/\/devblogs.microsoft.com\/dotnet\/wp-content\/uploads\/sites\/10\/2024\/12\/10AzureAISearchIndexFields.png\")
<\/p>\nWhy Azure AI Search?<\/h3>\n
Explaining the Code<\/h2>\n
Console Applications for Demonstrations<\/h3>\n
![\"Visual](\"https:\/\/devblogs.microsoft.com\/dotnet\/wp-content\/uploads\/sites\/10\/2024\/12\/20SolutionWithAllTheSamples.png\")
<\/p>\nShared Class for Data Representation<\/h3>\n
\n
Handling Different Data Types for Keys<\/h3>\n
\n
ulong<\/code> as the data type for its key field.<\/li>\nstring<\/code> as the key field\u2019s data type.<\/li>\nMovie Factory Implementation<\/h4>\n
public class MovieFactory<T>\n{\n public static List<Movie<T>> GetMovieList()\n {\n var movieData = new List<Movie<T>>()\n {\n \/\/ all movie sample collection is defined here\n };\n return movieData;\n }\n\n public static List<MovieVector<T>> GetMovieVectorList()\n {\n var movieData = GetMovieList();\n var movieVectorData = new List<MovieVector<T>>();\n foreach (var movie in movieData)\n {\n movieVectorData.Add(new MovieVector<T>\n {\n Key = movie.Key,\n Title = movie.Title,\n Description = movie.Description\n });\n }\n return movieVectorData;\n }\n<\/code><\/pre>\nWhat\u2019s Coming Next?<\/h2>\n
Conclusion and Learn More<\/h2>\n
Learn More<\/h3>\n
\n
Summary<\/h2>\n