Vector stores in Agent Framework for Python: familiar foundations, streamlined for agents

Vector stores have been part of the Python experience in Semantic Kernel for some time. With Microsoft Agent Framework, we have ported that work forward rather than starting over.

This post focuses on Python. For the .NET equivalent, see Use vector stores in .NET AI apps.

The foundation remains familiar: typed record definitions, stores and collections, batch storage and retrieval, embedding generation, vector search, and a growing set of integrations. If you have used Semantic Kernel’s vector-store APIs, you will recognize the overall model.

At the same time, this is more than a straight port. We streamlined model registration and the batch APIs, updated filters and embedding options, and made vector stores fit naturally into how agents receive and use context.

The most important additions are the context providers. They expose your data to agents through tools, or use a vector store to keep conversation history searchable without continually expanding the model context.

The vector-store APIs remain experimental, and the integration packages are alpha or beta. Use each implementation’s documentation for setup and its supported capabilities.

Define the data your application already owns

Agent Framework starts with your data model rather than a database SDK model. Mark the key, data, and vector fields with VectorStoreField, and @vectorstoremodel derives the collection definition:

from dataclasses import dataclass
from typing import Annotated

from agent_framework import (
    Filter,
    InMemoryCollection,
    VectorStoreField,
    vectorstoremodel,
)
from agent_framework.openai import OpenAIEmbeddingClient


@vectorstoremodel(collection_name="documents")
@dataclass
class Document:
    id: Annotated[str, VectorStoreField("key")]
    category: Annotated[str, VectorStoreField("data", is_indexed=True)]
    text: Annotated[str, VectorStoreField("data")]
    embedding: Annotated[
        list[float] | str | None,
        VectorStoreField(
            "vector",
            dimensions=1536,
            distance_function="cosine_similarity",
        ),
    ] = None


async def index_and_search() -> None:
    documents: InMemoryCollection[str, Document] = InMemoryCollection(
        Document, embedding_generator=OpenAIEmbeddingClient(model="text-embedding-3-small"),
    )
    await documents.ensure_collection_exists()

    await documents.upsert([
        Document(
            id="release-checklist",
            category="operations",
            text="Verify rollback, monitoring, and owner sign-off.",
            embedding="Verify rollback, monitoring, and owner sign-off.",
        )
    ])

    results = await documents.search(
        "What should I check before a release?",
        filter=Filter("category", "eq", "operations"),
        top=3,
    )

    async for result in results:
        print(result["record"].text, result["score"])

Set OPENAI_API_KEY before running the code. The records are stored in-memory (so this will not survive a restart), while OpenAI generates embeddings during both upsert and search. You can swap out the InMemoryCollection with any of the other implementations without changing anything else, see the list below for the ones we currently support.

Dataclasses are only one option. The same model layer supports Pydantic models, msgspec structs, plain classes, dictionaries, and externally owned types through custom codecs.

Turn vector search into an agent tool

As in Semantic Kernel, you can expose search directly as a first-class agent tool. create_vector_search_tool wraps a searchable collection and lets you control its name, description, result shape, filters, and approval behavior.

Param now makes custom search tools easier to build. Place a Param inside a Filter or FilterGroup, and the helper exposes it as a typed argument the model can supply. Given a collection configured to embed text queries, the tool definition stays small:

from typing import Literal

from agent_framework import Filter, Param, create_vector_search_tool

category = Param(
    "category",
    Literal["engineering", "operations", "product"],
    required=False,
    description="Only return documents from this category.",
)

search_documents = create_vector_search_tool(
    collection,
    name="search_documents",
    description="Search the document collection.",
    filter=Filter("category", "eq", category),
    result_mapper=lambda result: result["record"].text,
)

The resulting tool always accepts a natural-language query and optionally accepts category. The Literal values become an enum in the tool schema, giving the agent a clear set of valid choices. If the model omits the optional parameter, that filter is omitted. More complex tools can compose several parameters through FilterGroup.

The key addition: make collections available through context providers

When an agent needs more than one focused search tool, VectorCollectionContextProvider provides the higher-level experience.

VectorCollectionContextProvider can add upsert, get, delete, and search tools to an agent, together with instructions for using them. You can expose a read-only knowledge collection, let an agent manage records with approvals, scope the available data, or replace the default search with specialized tools.

Embedding setup stays specific to the operation: ingestion can use document embedding options while search uses query embedding options. Generated tools forward those settings to the same collection.

A context provider for searchable conversation history

Conversation histories keep growing, increasing the context sent to the model on every turn. VectorStoreHistoryProvider persists the full conversation in a searchable vector collection while loading only a compacted working set into the model context.

When include_search_tool=True, the agent also receives a tool for searching the complete scoped transcript. The active context can therefore remain bounded without losing access to an older decision, requirement, or user preference: the agent retrieves the relevant part of the conversation when it needs it instead of carrying every prior message on every invocation.

Embeddings and search without hard-coding a provider

Applications can supply precomputed vectors or have Agent Framework call an embedding service during upsert and search. They can generate every vector field or only selected fields, and pass different provider options for document ingestion and query generation. Vector dimensions stay with the model definition, so Agent Framework can validate generated and supplied vectors before they reach the backing store.

Search uses the same collection API across integrations, with portable data filters and keyword-hybrid search where supported. Each implementation documents its supported indexes, filters, metrics, and search modes.

Available integrations

The current Python integrations are:

• Azure AI Search
• Azure Cosmos DB for NoSQL
• Azure DocumentDB
• MongoDB
• Oracle AI Database 26ai
• Redis
• SQL Server and Azure SQL
• PostgreSQL with pgvector
• DuckDB
• Qdrant
• The core in-memory store for tests and local development

The same model and core operations work across integrations, while each database keeps its own features and constraints.

Get started

Start with the runnable vector-store samples for models, search, and agent tools. The vector-store-backed conversation sample demonstrates bounded working context with search over the full conversation.

Install the integration package for the store you want to use, and use its README for setup and supported capabilities. We would especially like feedback on the context-provider experience, searchable history, and which integrations you need next. Open an issue in the Microsoft Agent Framework repository and tell us what you are building.