{"id":973,"date":"2025-05-13T21:18:06","date_gmt":"2025-05-13T21:18:06","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/all-things-azure\/?p=973"},"modified":"2025-05-13T22:25:14","modified_gmt":"2025-05-13T22:25:14","slug":"accelerate-ai-applications-with-semantic-caching-on-azure-managed-redis","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/all-things-azure\/accelerate-ai-applications-with-semantic-caching-on-azure-managed-redis\/","title":{"rendered":"Accelerate AI Applications with Semantic Caching on Azure Managed Redis"},"content":{"rendered":"

In this blog, we will look at implementing the Semantic caching use case using Azure Managed Redis<\/a>. Azure has been at the forefront of providing caching solutions for over a decade with the Azure Cache for Redis enterprise. This service empowered developers with a high-performance, scalable cache that significantly enhanced the responsiveness of their applications. Now, Azure has taken another significant step forward in this area with its latest offering, AMR, which is currently in public preview. AMR runs on the Redis enterprise stack which offers significant advantages over the community edition of Redis. AMR provides advanced features such as Active Geo-Replication, Vector Storage & Search, Semantic Caching, Automatic Zone Redundancy, and Entra ID auth support across all its SKUs and Tiers.<\/p>\n

Why semantic caching?<\/h2>\n

Redis use cases have been expanding over the years from the traditional Data cache, API Response cache, Session Store, etc. to now include AI Apps use cases like Vector Store, Vector Search, and Semantic cache<\/a>. Using LLMs often introduces a high amount of latency (due to generation time) and cost (due to per token pricing) to an application. Semantic Caching can help solve these problems by storing the past output of an LLM along with the text as well as vector representation of the query associated with the LLM Output. The subsequent call to LLM can be preceded by cache lookup using the vector representation of the query; this pattern fits very well with the AI Apps implementations.<\/p>\n

Some common scenarios to use semantic caching could be in Faster FAQ retrieval in a chat bot by doing vector search of user query in the cache before making a call to the LLM Completion API and Storing previous interactions of users and their context to provide more relevant and personalized responses in a much faster time compared to relying on LLM for generating completion response all the time.<\/p>\n

Let\u2019s look at the architecture of an AI App implementing the Semantic caching pattern:<\/p>\n

 <\/p>\n

\"redis<\/a><\/p>\n

As depicted above, the AI App looks up the semantic cache (AMR) before invoking the LLM chat completion API. The cache lookup is not text based alone, the AI App gets a vector representation of the query by invoking the LLM embedding API, this enables semantic search of the user queries. Apart from the AI App, Semantic caching logic can also be implemented in API Management (APIM) when using the AI Gateway pattern. APIM provides built in semantic caching policies<\/a> that makes implementing this use-case very straightforward.<\/p>\n

Example: Python App with Semantic Cache<\/h2>\n

Let\u2019s now look at a Python application sample that uses semantic caching pattern and the improvement in response times achieved with this.<\/p>\n

In this section of the app, we import the required libraries and configure API credentials to work with Azure OpenAI and Azure Redis:<\/p>\n

\"REDIS<\/a><\/p>\n

Next, we will create a Semantic Cache Index in Azure Redis. Note that Azure OpenAI Embedding API<\/a> is being set as the embedding provider for this Semantic Cache instance. Another important config here is the Semantic Cache Distance Threshold, the lower the value, the more similar the cached and new queries must be to be considered a match in the cache, valid values for this parameter are between 0 and 1.<\/p>\n

\"REDIS<\/a><\/p>\n

The last section of the app includes code to get a prompt from the user, checking for the presence of similar prompt\/query from the user in the semantic cache. If there is a match, the response is returned from the Cache and the call to Chat Completion operation in Azure OpenAI is skipped. If there is no match in the cache, then the Chat Completion operation is invoked, and the response is stored in the semantic cache index of Azure Redis.<\/p>\n

\"REDIS<\/a><\/p>\n

To see this in action, we run the application and pass the following query to the application:<\/p>\n

\"REDIS<\/a><\/p>\n

As expected, the cache lookup resulted in a miss and Azure OpenAI was called to get the response, and it took more than 3 seconds for the execution to complete:<\/p>\n

\"REDIS<\/a><\/p>\n

We now run the application again by passing a different query:<\/p>\n

\"REDIS<\/a><\/p>\n

Although the words of this query are different, the cache search led to a hit as the query was semantically like the first query. The response time was less than 200 milliseconds, which is more than 10 times faster than calling LLM.<\/p>\n

\"REDIS<\/a><\/p>\n

As depicted above, using Semantic Search capabilities of Azure Managed Redis where applicable, will lead to better latency and reduced costs for your intelligent applications.<\/p>\n

Resources<\/h2>\n