{"id":4940,"date":"2025-06-24T17:40:21","date_gmt":"2025-06-25T00:40:21","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/semantic-kernel\/?p=4940"},"modified":"2025-06-26T00:27:05","modified_gmt":"2025-06-26T07:27:05","slug":"semantic-kernel-python-gets-a-major-vector-store-upgrade","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/agent-framework\/semantic-kernel-python-gets-a-major-vector-store-upgrade\/","title":{"rendered":"Semantic Kernel Python Gets a Major Vector Store Upgrade"},"content":{"rendered":"

We’re excited to announce a significant update to Semantic Kernel Python’s vector store implementation. Version 1.34 brings a complete overhaul that makes working with vector data simpler, more intuitive, and more powerful. This update consolidates the API, improves developer experience, and adds new capabilities that streamline AI development workflows.<\/p>\n

What Makes This Release Special?<\/h2>\n

The new vector store architecture consolidates everything under semantic_kernel.data.vector<\/code> and delivers three key improvements:<\/p>\n

    \n
  1. Simplified API<\/strong>: One unified field model replaces multiple complex field types<\/li>\n
  2. Integrated Embeddings<\/strong>: Embedding generation happens automatically where you need it<\/li>\n
  3. Enhanced Features<\/strong>: Advanced filtering, hybrid search, and streamlined operations<\/li>\n<\/ol>\n

    Let’s explore what makes these changes valuable.<\/p>\n

    Unified Field Model – Simplified Configuration<\/h2>\n

    We’ve replaced three separate field types with one powerful VectorStoreField<\/code> class that handles everything you need.<\/p>\n

    Before: The Old Way (Complex and Verbose)<\/h3>\n
    from semantic_kernel.data import (\r\n    VectorStoreRecordKeyField,\r\n    VectorStoreRecordDataField, \r\n    VectorStoreRecordVectorField\r\n)\r\n\r\n# Multiple classes to remember and configure\r\nfields = [\r\n    VectorStoreRecordKeyField(name=\"id\"),\r\n    VectorStoreRecordDataField(name=\"text\", is_filterable=True, is_full_text_searchable=True),\r\n    VectorStoreRecordVectorField(name=\"vector\", dimensions=1536, distance_function=\"cosine\")\r\n]\r\n<\/code><\/pre>\n
    After: The New Way (Clean and Intuitive)<\/h3>\n
    from semantic_kernel.data.vector import VectorStoreField\r\nfrom semantic_kernel.connectors.ai.open_ai import OpenAITextEmbedding\r\n\r\nembedding_service = OpenAITextEmbedding(ai_model_id=\"text-embedding-3-small\")\r\n\r\n# One class handles all field types\r\nfields = [\r\n    VectorStoreField(\"key\", name=\"id\"),\r\n    VectorStoreField(\"data\", name=\"text\", is_indexed=True, is_full_text_indexed=True),\r\n    VectorStoreField(\"vector\", name=\"vector\", dimensions=1536, \r\n                    distance_function=\"cosine\", embedding_generator=embedding_service)\r\n]\r\n<\/code><\/pre>\n
    This approach provides cleaner code with better IDE support, including improved autocomplete and clearer intentions.<\/p>\n
    Integrated Embeddings – Automatic Generation<\/h2>\n
    The new architecture includes automatic embedding generation directly in your field definitions. No more manual embedding steps\u2014just define what you want embedded, and it happens automatically.<\/p>\n
    from semantic_kernel.data.vector import VectorStoreField, vectorstoremodel\r\nfrom semantic_kernel.connectors.ai.open_ai import OpenAITextEmbedding\r\nfrom typing import Annotated\r\nfrom dataclasses import dataclass\r\n\r\n@vectorstoremodel\r\n@dataclass\r\nclass MyRecord:\r\n    content: Annotated[str, VectorStoreField('data', is_indexed=True, is_full_text_indexed=True)]\r\n    title: Annotated[str, VectorStoreField('data', is_indexed=True, is_full_text_indexed=True)]\r\n    id: Annotated[str, VectorStoreField('key')]\r\n    vector: Annotated[list[float] | str | None, VectorStoreField(\r\n        'vector', \r\n        dimensions=1536, \r\n        distance_function=\"cosine\",\r\n        embedding_generator=OpenAITextEmbedding(ai_model_id=\"text-embedding-3-small\"),\r\n    )] = None\r\n\r\n    def __post_init__(self):\r\n        if self.vector is None:\r\n            # Combine multiple fields for richer embeddings\r\n            self.vector = f\"Title: {self.title}, Content: {self.content}\"\r\n<\/code><\/pre>\n
    You can now easily combine multiple fields to create richer embeddings with simple field assignment.<\/p>\n
    Lambda-Powered Filtering – Type-Safe and Expressive<\/h2>\n
    The new filtering system uses lambda expressions that are type-safe, IDE-friendly, and highly expressive, replacing the previous string-based FilterClause<\/code> objects.<\/p>\n
    Before: String-Based Complexity<\/h3>\n
    from semantic_kernel.data.text_search import SearchFilter\r\n\r\n# Multiple objects and method calls\r\ntext_filter = SearchFilter()\r\ntext_filter.equal_to(\"category\", \"AI\")\r\ntext_filter.equal_to(\"status\", \"active\")\r\n<\/code><\/pre>\n
    After: Lambda Expression Power<\/h3>\n
    # Clean, readable, and type-safe\r\nresults = await collection.search(\r\n    \"query text\", \r\n    filter=lambda record: record.category == \"AI\" and record.status == \"active\"\r\n)\r\n\r\n# Complex filtering with multiple conditions\r\nresults = await collection.search(\r\n    \"machine learning concepts\",\r\n    filter=lambda record: (\r\n        record.category == \"AI\" and \r\n        record.score > 0.8 and\r\n        \"important\" in record.tags and\r\n        0.5 <= record.confidence_score <= 0.9\r\n    )\r\n)\r\n<\/code><\/pre>\n
    Your IDE can now provide full autocomplete support and catch errors at development time.<\/p>\n
    Streamlined Operations – Consistent Interface<\/h2>\n
    The new API provides a consistent interface that works with both single records and batches:<\/p>\n
    from semantic_kernel.connectors.in_memory import InMemoryCollection\r\n\r\ncollection = InMemoryCollection(\r\n    record_type=MyRecord,\r\n    embedding_generator=OpenAITextEmbedding(ai_model_id=\"text-embedding-3-small\")\r\n)\r\n\r\n# Single record or batch - same method\r\nawait collection.upsert(single_record)\r\nawait collection.upsert([record1, record2, record3])\r\n\r\n# Flexible retrieval\r\nawait collection.get([\"id1\", \"id2\"])  # Get specific records\r\nawait collection.get(top=10, skip=0, order_by='title')  # Browse with pagination\r\n\r\n# Powerful search with automatic embedding\r\nresults = await collection.search(\"find AI articles\", top=10)\r\nresults = await collection.hybrid_search(\"machine learning\", top=10)\r\n<\/code><\/pre>\n
    Instant Search Functions – Simplified Creation<\/h2>\n
    Creating search functions for your kernel is now straightforward:<\/p>\n
    Before: Multiple Steps and Setup<\/h3>\n
    from semantic_kernel.data import VectorStoreTextSearch\r\n\r\ncollection = InMemoryCollection(collection_name='collection', record_type=MyRecord)\r\nsearch = VectorStoreTextSearch.from_vectorized_search(\r\n    vectorized_search=collection, \r\n    embedding_generator=OpenAITextEmbedding(ai_model_id=\"text-embedding-3-small\")\r\n)\r\nsearch_function = search.create_search(function_name='search')\r\n<\/code><\/pre>\n
    After: Streamlined Creation<\/h3>\n
    # Create a search function directly on your collection\r\nsearch_function = collection.create_search_function(\r\n    function_name=\"search\",\r\n    search_type=\"vector\",  # or \"keyword_hybrid\"\r\n    top=10,\r\n    vector_property_name=\"vector\"\r\n)\r\n\r\n# Add to kernel\r\nkernel.add_function(plugin_name=\"memory\", function=search_function)\r\n<\/code><\/pre>\n
    Enhanced Data Model Expressiveness<\/h2>\n
    The simplified API doesn’t sacrifice expressiveness. Data models are more capable than before:<\/p>\n
    @vectorstoremodel(collection_name=\"documents\")\r\n@dataclass\r\nclass DocumentRecord:\r\n    # Rich metadata\r\n    id: Annotated[str, VectorStoreField('key')]\r\n    title: Annotated[str, VectorStoreField('data', is_indexed=True, is_full_text_indexed=True)]\r\n    content: Annotated[str, VectorStoreField('data', is_full_text_indexed=True)]\r\n    category: Annotated[str, VectorStoreField('data', is_indexed=True)]\r\n    tags: Annotated[list[str], VectorStoreField('data', is_indexed=True)]\r\n    created_date: Annotated[datetime, VectorStoreField('data', is_indexed=True)]\r\n    confidence_score: Annotated[float, VectorStoreField('data', is_indexed=True)]\r\n    \r\n    # Multiple vectors for different purposes\r\n    content_vector: Annotated[list[float] | str | None, VectorStoreField(\r\n        'vector', \r\n        dimensions=1536,\r\n        storage_name=\"content_embedding\",\r\n        embedding_generator=OpenAITextEmbedding(ai_model_id=\"text-embedding-3-small\")\r\n    )] = None\r\n    \r\n    title_vector: Annotated[list[float] | str | None, VectorStoreField(\r\n        'vector',\r\n        dimensions=1536, \r\n        storage_name=\"title_embedding\",\r\n        embedding_generator=OpenAITextEmbedding(ai_model_id=\"text-embedding-3-small\")\r\n    )] = None\r\n\r\n    def __post_init__(self):\r\n        if self.content_vector is None:\r\n            self.content_vector = self.content\r\n        if self.title_vector is None:\r\n            self.title_vector = self.title\r\n<\/code><\/pre>\n
    Better Connector Experience<\/h2>\n
    We’ve also streamlined the connector imports and naming. Everything is now logically organized under semantic_kernel.connectors<\/code>:<\/p>\n
    # Clean, consistent imports\r\nfrom semantic_kernel.connectors.azure_ai_search import AzureAISearchStore\r\nfrom semantic_kernel.connectors.chroma import ChromaVectorStore\r\nfrom semantic_kernel.connectors.pinecone import PineconeVectorStore\r\nfrom semantic_kernel.connectors.qdrant import QdrantVectorStore\r\n\r\n# Or use the convenient lazy loading\r\nfrom semantic_kernel.connectors.memory import (\r\n    AzureAISearchStore,\r\n    ChromaVectorStore,\r\n    PineconeVectorStore,\r\n    QdrantVectorStore\r\n)\r\n<\/code><\/pre>\n
    Real-World Example: Complete Implementation<\/h2>\n
    Here’s how a complete example looks with the new architecture:<\/p>\n
    The New Way (Simple and Powerful)<\/h3>\n
    from semantic_kernel.data.vector import VectorStoreField, vectorstoremodel\r\nfrom semantic_kernel.connectors.in_memory import InMemoryCollection\r\nfrom semantic_kernel.connectors.ai.open_ai import OpenAITextEmbedding\r\nfrom typing import Annotated\r\nfrom dataclasses import dataclass\r\n\r\n@vectorstoremodel(collection_name=\"knowledge_base\")\r\n@dataclass\r\nclass KnowledgeBase:\r\n    id: Annotated[str, VectorStoreField('key')]\r\n    content: Annotated[str, VectorStoreField('data', is_full_text_indexed=True)]\r\n    category: Annotated[str, VectorStoreField('data', is_indexed=True)]\r\n    vector: Annotated[list[float] | str | None, VectorStoreField(\r\n        'vector', \r\n        dimensions=1536,\r\n        embedding_generator=OpenAITextEmbedding(ai_model_id=\"text-embedding-3-small\")\r\n    )] = None\r\n\r\n    def __post_init__(self):\r\n        if self.vector is None:\r\n            self.vector = self.content\r\n\r\n# Create collection with automatic embedding\r\nasync with InMemoryCollection(record_type=KnowledgeBase) as collection:\r\n    await collection.ensure_collection_exists()\r\n    \r\n    # Add documents (embeddings created automatically)\r\n    docs = [\r\n        KnowledgeBase(id=\"1\", content=\"Semantic Kernel is awesome\", category=\"general\"),\r\n        KnowledgeBase(id=\"2\", content=\"Python makes AI development easy\", category=\"programming\"),\r\n    ]\r\n    await collection.upsert(docs)\r\n    \r\n    # Search with intelligent filtering\r\n    results = await collection.search(\r\n        \"AI development\", \r\n        top=5,\r\n        filter=lambda doc: doc.category == \"programming\"\r\n    )\r\n    \r\n    # Create kernel search function\r\n    search_func = collection.create_search_function(\"knowledge_search\", search_type=\"vector\")\r\n    kernel.add_function(plugin_name=\"kb\", function=search_func)\r\n<\/code><\/pre>\n
    What This Means for Your Projects<\/h2>\n
    This update brings several concrete benefits:<\/p>\n