My team recently had the opportunity to build a project using Azure Cosmos DB with integrated cache. We worked on a customer engagement for a use case that involved heavy reads, low writes, and repeated queries. For our backing datastore we used Azure Cosmos DB, a horizontally-scaled document database. We decided to try out the integrated cache feature because it looked like it might be a good fit for our customer’s use case. We spoke with members of Azure Cosmos DB product team to gain a better understanding of how the integrated cache operates and we applied those learnings to our project.<\/p>\n
What is Azure Cosmos DB with integrated cache?<\/h2>\n
The Azure Cosmos DB integrated cache<\/a> is essentially a turn-key cache that is managed for you. While Azure Cosmos DB does support multiple APIs, the integrated cache is currently only available for the NoSQL API<\/a>. With the integrated cache you don\u2019t need to set up any additional caching resources or dependencies and your code can simply interact with Cosmos DB without having to explicitly manage the cache.<\/p>\n When you create your Azure Cosmos DB account and enable the dedicated gateway<\/a>, the integrated cache is configured automatically. As part of the process to provision the dedicated gateway<\/a>, you select the number of gateway nodes you\u2019d like to create. One node is sufficient for development purposes, but for use in production it is recommended to have a minimum of three. The cost for the dedicated gateway<\/a> is calculated hourly and is determined by the number of nodes and the region selected. Since queries that hit the integrated cache incur no request unit (RU) charges, the overall cost of the database usage will be primarily driven by this hourly charge.<\/p>\n Because the integrated cache is managed for you, it does not give you explicit controls over its operations the way a stand-alone cache would. Therefore it is best suited for applications that:<\/p>\n Let’s take a global e-commerce platform as an example. Such applications are predominantly read-heavy. Users frequently view product listings, but the underlying product information doesn’t change very often. Because these types of queries are recurrent, the low latency offered by Azure Cosmos DB with Integrated Cache can considerably enhance the user experience. The eventual or session consistency models work well in this scenario because users don’t need immediate updates on slight changes such as new product reviews. Moreover, as these applications don’t need explicit control over cache refreshing, the automatic background refreshes provided by Azure Cosmos DB with Integrated Cache can maintain a high-quality user experience even in the face of occasional minor data discrepancies.<\/p>\n When an application sends a request to Azure Cosmos DB using the dedicated gateway, this request first reaches a dedicated gateway node. This node checks its cache for the requested data. If the data isn’t present in the cache (a “cache miss”), the gateway node communicates with the Azure Cosmos DB database to fetch the necessary data. After fetching the data from the database, the gateway node stores a copy of it in its own cache for future requests and then finally sends the data back to the original caller. Each cache miss requires data retrieval from the database node, which results in request unit charges.<\/p>\n Each dedicated gateway node maintains its own cache<\/a>. This means you can potentially get back different data depending on which node the request is routed through (see the docs<\/a> and the animation below). The backing data in the database is replicated as usual across all the backend nodes, but caches are not replicated across dedicated gateway nodes. Calls are routed to dedicated gateway nodes randomly, so it is possible to have a cache miss even if the data is cached on another dedicated gateway node.<\/p>\n Azure Cosmos DB integrated cache has an item cache and a query cache:<\/p>\n Azure Cosmos DB with integrated cache handles cache refresh differently from other caching technologies. At this time there is no explicit cache invalidation, but that may be included in a future release. A workaround for if you need to invalidate all cached data on all dedicated gateway nodes is to deprovision and re-provision the dedicated gateway.<\/p>\n As a cache on a given dedicated gateway node fills up with data, it evicts the least-recently accessed values. The item cache is automatically “refreshed” when an item is newly created, updated, or deleted. That is not true for values in the query cache.<\/p>\n The primary mechanism to trigger a query cache refresh is via the Be aware that queries that return no results are cached. This is an edge case in which an application queries data that does not yet exist, the data is then inserted into the database, and the same query is run again. We encountered this case during testing for our project. As a result we decided to skip the cache for certain types of queries in which the backing data changes more frequently. To always hit the database, you can connect to Cosmos DB via direct mode<\/a> even with an account that has a dedicated gateway.<\/p>\n Below is an animation of what happens when a request is sent to the dedicated gateway endpoint.<\/p>\n You can download and walk through the code for my working sandbox project that uses Azure Cosmos DB with integrated cache. Start with the README<\/a> to understand how to set up for local development.<\/p>\n To configure a To build a query that hits the query cache, use the To read from the item cache, send a The document will be returned in the The simplification of application architecture is what we liked the most about Azure Cosmos DB with integrated cache. It’s convenient to be able to simply write Cosmos queries and know that cache hits and misses are handled automatically. For usage patterns involving a high volume of repeated<\/em> requests, Azure Cosmos DB with integrated cache can serve thousands of requests per second without a corresponding high Request Unit charge. The dedicated gateway costs can be predicted in advance, which can result in overall lower Cosmos DB charges as compared to a Azure Cosmos DB without<\/em> an integrated cache. We look forward to additional features the Azure Cosmos DB product team may develop to make this an even more robust product.<\/p>\n I want to give thanks to crew members Bindu Chinnasamy<\/a>, John Hauppa<\/a>, Cameron Taylor<\/a>, and Kanishk Tantia<\/a>. I also want to thank Justine Cocchi<\/a> and Madhav Annamraju<\/a> who provided information and resources to help us understand this feature better.<\/p>\nWhen is Azure Cosmos DB with Integrated Cache a good fit?<\/h2>\n
\n
Architecture of Azure Cosmos DB with Integrated Cache<\/h2>\n
\n
key<\/code> is a composite of the item id and partition key and the value<\/code> is the data in the associated document. The item cache works like write-through cache, which means that the data in the cache is updated at the time it is written.<\/li>\nWHERE clause<\/code> will be cached in the query cache even if it returns a single value. The key<\/code> for an item in the query cache is based on the query text and the value<\/code> is the result of that query. Even if a particular document has been cached previously as part of the result for another query, it will be cached again if a given query varies from the previous one. For example, a request that returns documents [1..40] will be cached as one query result and a request that returns documents [2..41] will be cached as a distinct query result. Changes to the backing data do not affect cached values in the query cache.<\/li>\n<\/ul>\nCache Refresh<\/h2>\n
MaxIntegratedCacheStaleness<\/code> value that is passed in when an application requests data from Cosmos DB. The calling application essentially tells the dedicated gateway what its tolerance is for stale data. That tolerance may be mere minutes to several days depending on the application\u2019s use cases for the data. When the MaxIntegratedCacheStaleness<\/code> value exceeds the amount of time that has passed since a query result was cached, the query is run against the database again and the updated result is stored in the cache on that dedicated gateway node. If an application queries Cosmos with a MaxIntegratedCacheStaleness<\/code> of 0, it will always get back data from the database and the cache on that dedicated gateway node will be refreshed to store the new query results. The caches on any other dedicated gateway nodes will not be refreshed until a request that \u201cexpires\u201d the cached values is routed to that node or it is evicted according to the Least Recently Used (LRU) eviction policy.<\/p>\nAzure Cosmos DB with Integrated Cache Request Routing and Cache Refresh<\/h2>\n
\n
MaxIntegratedCacheStaleness<\/code> value sent with the request, no call is sent to a backend database node. The cached data is returned.<\/li>\nMaxIntegratedCacheStaleness<\/code> value sent with the request, the data is retrieved from a backend database node, cached in the database, and then returned.<\/li>\n<\/ul>\n![\"Animation](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2023\/08\/CosmosRequestRouting.gif\")
<\/p>\nCode Examples<\/h2>\n
Configure CosmosClient to Use Dedicated Gateway<\/h3>\n
CosmosClient<\/code> to use your dedicated gateway, you will need to supply your dedicated gateway connection string as well as CosmosClientOptions<\/code> that specify the ConnectionMode<\/code> and ConsistencyLevel<\/code> for the client to use.<\/p>\n var dedicatedConnectionString = \"AccountEndpoint=https:\/\/<cosmos-db-account-name>.sqlx.cosmos.azure.com\/;AccountKey=<cosmos-db-account-key>;\";\r\n var dedicatedGatewayCosmosClient = new CosmosClient(dedicatedConnectionString,\r\n new CosmosClientOptions()\r\n {\r\n ConnectionMode = ConnectionMode.Gateway,\r\n ConsistencyLevel = ConsistencyLevel.Eventual\r\n });<\/code><\/pre>\nHit the Integrated Query Cache<\/h3>\n
GetItemLinqQueryable<\/code> method on your CosmosContainer<\/code> object. In your QueryRequestOptions<\/code> add the value you wish to use for MaxIntegratedCacheStaleness<\/code> and set your ConsistencyLevel<\/code>.<\/p>\n var sortedRefIds = refIds.OrderBy(id => id).ToList(); \/\/ Always sending a request in a specific order will maximize your chances of a cache hit\r\n\r\n IQueryable<TestItem> queryable = containerWithDedicatedConnection.GetItemLinqQueryable<TestItem>(\r\n requestOptions: new QueryRequestOptions\r\n {\r\n DedicatedGatewayRequestOptions = new DedicatedGatewayRequestOptions()\r\n {\r\n MaxIntegratedCacheStaleness = TimeSpan.FromHours(cacheStaleness)\r\n },\r\n ConsistencyLevel = ConsistencyLevel.Eventual\r\n }\r\n );\r\n\r\n queryable = queryable.Where(e => sortedRefIds.Contains(e.RefId));\r\n\r\n using (var linqFeed = queryable.ToFeedIterator())\r\n {\r\n while (linqFeed.HasMoreResults)\r\n {\r\n var feedResponse = await linqFeed.ReadNextAsync();\r\n\r\n result.RequestUnits = feedResponse.RequestCharge; \/\/ When this has a value of 0, that means you've successfully hit the cache\r\n\r\n foreach (var item in feedResponse)\r\n {\r\n testItems.Add(item);\r\n }\r\n }\r\n }<\/code><\/pre>\nHit the Item Cache<\/h3>\n
ReadItemAsync<\/code> request along with ItemRequestOptions<\/code> that point the request to use the dedicated gateway.<\/p>\n ItemResponse<TestItem> response = await containerWithDedicatedConnection.ReadItemAsync<TestItem>(id, new PartitionKey(partitionKey), \r\n new ItemRequestOptions() \r\n {\r\n DedicatedGatewayRequestOptions = new DedicatedGatewayRequestOptions()\r\n {\r\n MaxIntegratedCacheStaleness = TimeSpan.FromHours(cacheStaleness)\r\n },\r\n ConsistencyLevel = ConsistencyLevel.Eventual\r\n });\r\n\r\n var testItemQueryResult = new TestItemQueryResult();\r\n\r\n testItemQueryResult.TestItems.Add(response.Resource); \/\/ the document\r\n testItemQueryResult.RequestUnits = response.RequestCharge; \/\/ the request unit charge\r\n\r\n return testItemQueryResult;<\/code><\/pre>\nResource<\/code> property on the ItemResponse<\/code> object and the number of Request Units consumed will be returned on the RequestCharge<\/code> property.<\/p>\nConclusion<\/h2>\n
Acknowledgements<\/h2>\n