In the first part of our series on Azure Cosmos DB Vector Search with DiskANN<\/a>, we explored the fundamentals of vector indexing and search capabilities using Azure Cosmos DB for NoSQL and demonstrated a few early performance and cost characteristics. In Part 2, we\u2019ll demonstrate how to scale to 1 billion vector datasets, while maintaining a query latency of under 100 milliseconds with > 90% recall, all in a cost-effective manner. <\/strong>Along the way, we will review concepts like partitioning for managing large-scale vector data in Azure Cosmos DB, along with optimizations & best practices for faster ingestion and reducing cross-partition query latencies. You’ll learn to design, optimize and measure large-scale vector indexes in Azure Cosmos DB for efficiency and cost-effective. Let’s dive in!<\/p>\n You can find the code to replicate the experiments in this article on GitHub: VectorIndexScenarioSuite<\/a>. Refer to the MSTuringEmbeddingOnlyScenario.cs<\/span><\/a> file, for a holistic view of the operations. We also recommend that you familiarize yourself with creating vector indexes in Azure Cosmos DB for NoSQL by reviewing part 1 of this series<\/a>.<\/p>\n Azure Cosmos DB <\/span>for NoSQL <\/span><\/span>uses partitioning <\/span><\/span><\/span>to ensure <\/span>low-latency<\/span> across large indices. <\/span><\/span><\/span>Internally, it<\/span><\/span><\/span> utilizes a single DiskANN instance per physical partition, which allows for faster indexing as each physical partition can use its own RU (Request Unit) allotment to independently index vectors. The number of physical partitions in a container depends on the following characteristics:<\/p>\n Azure Cosmos DB for NoSQL has logical partitions<\/em><\/a> that are defined by partition keys<\/em>. Many logical partitions are mapped to a single physical partition, which are units of data management.<\/p>\n Configuring the container with a throughput of N x 10,000 RU\/s, it creates N physical partitions. For example, if you create a collection with 50 x 10,000 RU\/s = 500,000 RU\/s, it will create 50 physical partitions. This setup can significantly increase indexing speeds because it distributes the work for indexing across all physical partitions.<\/p>\n When you choose a partition key, ensure it has high cardinality, meaning it offers a wide range of distinct values to evenly distribute data and RU usage across partitions, and create many logical partitions. For large-scale datasets, consider using hierarchical partition keys to create multi-level partitioning. You need to thoughtfully design hierarchical partition keys to ensure the first level maintains high cardinality, maximizing scalability and performance. You can learn more about partitioning in Azure Cosmos DB from our official documentation<\/a>.<\/p>\n In the experiments here, we use the vector ID as the partition key. This ensures a uniform distribution of data across partitions, which helps avoid hotspots and maintain efficiency.<\/p>\n Let\u2019s see how to batch documents to increase indexing throughput and reduce cross-partition query latency by increasing concurrency of the SDK.<\/p>\n The Azure Cosmos DB API allows us to insert multiple documents with vectors in one transaction. This helps lower the number of round trips between your client SDK and the Azure Cosmos DB servers. This also allows the vector indexer to parallelize the indexing of all vectors in the batch. This feature can enabled by specifying the AllowBulkExecution<\/em> option in the Azure Cosmos DB for NoSQL .NET<\/a>, Java<\/a>, and JavaScript<\/a> SDKs. The Python SDK<\/a> supports transactional batch, but doesn\u2019t support bulk execution at this time. However, you can use the async client<\/a> for concurrent requests. The batches that the SDK creates to optimize throughput have a current maximum of 100 operations per batch.<\/p>\n Here is an example of enabling AllowBulkExecution using the .NET SDK.<\/p>\n Low-latency querying is essential for developing responsive AI applications. In a scale-out configuration, the client SDK will query multiple physical partitions to expedite response times and improve latency. The Azure Cosmos DB SDK provides MaxConcurrency<\/em> option to control the number of concurrent operations run client-side during parallel query execution. If you set it to less than 0, the system automatically decides the number of concurrent operations to run. By setting a positive value for MaxConcurrency<\/em>, you can increase the number of concurrent operations to the specified value and reduce latency.<\/p>\n A few best practices to consider to optimize latency:<\/p>\n Now that we’ve covered key concepts, tips, and best practices for large-scale vector indexing and search, let’s put them into action. We’ll use the VectorIndexScenarioSuite<\/a> code, which makes it easy to run\u00a0experiments, benchmark performance, and validate the impact of different configuration choices, such as partitioning strategy, indexing parameters, and query patterns.<\/p>\n We will experiment with a billion-vector index dataset to illustrate the impact of partitioning on query latency (P50, P95, average), recall and RU charges. Note that recall@k measures the overlap between the top-k results returned by the index and actual top-k results \u2013 essentially the accuracy of the index.<\/p>\n We use the Microsoft Turing 1B dataset which consists of 1 billion Bing search queries encoded by Turing AGI v5. It can be downloaded with this script: ms_turing_download.ps1<\/a>. You could use an alternate dataset in your experiment, including those with 1000+ dimensions. Cosmos DB costs and query latency scales highly sublinearly with the number of dimensions as well.<\/p>\n We will manually configure the Azure Cosmos DB container with different RU settings to create different physical partitions. This allows us to observe their impact on RU consumption, query latency, and recall. The RU settings will be:<\/p>\n The latency measurements will include:<\/p>\n Latency numbers are commonly reported at different percentiles for a clear picture of overall distribution. The recall and latency distribution for top-K (K=10 here) vector search queries on post warmup are listed below. L is the search list parameters for DiskANN and equal to K*searchListSizeMultiplier, an optional parameter in the VectorDistance<\/a> query system function, set to 10 by default.<\/p>\n (Please note that these numbers are rounded and may improve in the future)<\/em><\/p>\nCode<\/h2>\n
Partitions and scale out<\/h2>\n
\n
Partition Key<\/h4>\n
Optimizations for large scale vector workloads<\/h2>\n
Faster Ingestion<\/h3>\n
CosmosClientOptions cosmosClientOptions = new()\r\n{\r\n ConnectionMode = ConnectionMode.Direct,\r\n AllowBulkExecution = true,\r\n \/\/ SDK will handle throttling and retry after recommended time has elapsed.\r\n \/\/ https:\/\/learn.microsoft.com\/en-us\/azure\/cosmos-db\/nosql\/how-to-migrate-from-bulk-executor-library\r\n MaxRetryAttemptsOnRateLimitedRequests = 100,\r\n MaxRetryWaitTimeOnRateLimitedRequests = TimeSpan.FromSeconds(600)\r\n};<\/code><\/pre>\nImproving Query Latency<\/h3>\n
var requestOptions = new QueryRequestOptions()\r\n{\r\n MaxConcurrency = 64,\r\n};\r\nFeedIterator<IdWithSimilarityScore> queryResultSetIterator =\r\nthis.CosmosContainerForQuery.GetItemQueryIterator<IdWithSimilarityScore>(queryDefinition,null, requestOptions);<\/code><\/pre>\n\n
Experiments and Evaluation<\/h2>\n
Experimental Setup<\/h3>\n
RU Charges Configuration<\/h3>\n
\n
Latency Comparison<\/h3>\n
\n