Introduction

In October 2024 Microsoft replaced a legacy rule‑based chat bot on Microsoft Store with Microsoft Store Assistant, powered by Azure Open AI, Semantic Kernel, and real‑time page context. The transformation changed a scripted, button-driven experience into a conversation that comprehends the entire public Microsoft portfolio, including Surface and Xbox products, Microsoft 365 subscriptions, Azure services, and the Dynamics and Power Platform portfolio, and knows when to involve a human Sales Associate. Six months later, the assistant manages several millions of conversations annually, maintains a four-star satisfaction rating, and generates revenue exceeding 140 percent of its forecast.

The challenge before October 2024

The previous solution relied on a rule‑based bot that steered shoppers through rigid decision trees and routed them to live agents via button clicks. Maintaining that framework was costly: each content update required manual topic management with answers, and every change spawned more flows to maintain. Human agents were inundated with off‑topic or incomplete chats because the bot could not reason over Microsoft’s vast and constantly changing web presence, hundreds of thousands of pages spanning consumer and commercial offerings, hardware, software, and cloud services. Iteration of the legacy bot lagged behind product launches and new information, customers rated the bot poor on CSAT measures, and operational expenses of the bot and for human agents were unsustainable.

From launch to current state

At launch, Microsoft Store Assistant utilized Azure Open AI gpt-4o with four ‘experts’ defined as Semantic Kernel skills, with streaming responses to the frontend. In subsequent months, the team upgraded model versions of gpt-4o through a simple configuration switch in Semantic Kernel and Azure Open AI, while automated simulations and evaluations with AI Foundry demonstrated functional and safety performance. New experts were added for tasks such as identifying appropriate phone numbers for customer requests, facilitating additional human transfers, and managing conversation closure. Real-time page context was integrated, fetching and parsing top-ranked Microsoft pages relevant to the discussion, which provided real-time detailed intelligence, along with stock and price data from Azure AI Search reasoning over the product catalog. Additional features included out-of-the-box solutions like Azure Open AI prompt caching, Azure Content Safety, and new features from Azure AI Foundry, all contributing to a highly performant, practical and maintainable assistant for customers.

Multi‑expert orchestration with Semantic Kernel

Store Assistant uses a multi-expert type of orchestration workflow in Semantic Kernel. There is a main ‘Coordinator’ that works like a planner and it decides which experts to invoke for each conversational turn. Each expert leverages a defined set of enrichment plug-ins giving it additional context to address the current conversation turn. This approach consistently provides the exact tool calls and context desired, with extremely low deviation. It also allows for the Coordinator to ask clarifying questions to improve hit rates of which expert and queries to invoke to improve the answer.

Coordinator agent

Semantic Kernel’s planner runs a comprehensive system prompt that:

• Reviews chat history, latest user message, and current page context (what page the customer is on, etc.).
• Applies a shared set of rules to follow as part of its personality and desired functions.
• Selects one expert skill or asks a clarifying question.

 Evaluation and safety at scale

When Store Assistant first launched, the team had to maintain bespoke simulation and evaluation tooling to run and evaluate thousands of simulated conversations for functional and safety performance. With the recent introduction of Azure AI Foundry, the entire product team are able to add/edit test cases and run thousands of synthetic conversations for every release, scoring relevance, groundedness, coherence, and other out of box metrics as well as custom evaluators measuring additional aspects of the AI solution. Results are stored centrally in the AI Foundry portal for product managers and engineers, make it easy to compare runs, and the data form part of each Deployment Safety Board submission as part of Responsible AI release mechanisms.

Conversation Analysis pipeline

In addition to the pre-launch evaluations, Store Assistant leverages online evaluations using Azure Open AI (o3-mini) and the native conversation storage in Azure CosmosDB with its change feed triggers.

1. Every closed chat is written to Azure Cosmos DB.
2. The change‑feed triggers an Azure Functions app.
3. The function uses Azure Open AI o3‑mini to classify topic, product, sentiment, resolution, and to redact any personal information detected by Azure Content Safety.
4. Analysis results are stored back in Cosmos DB and visualized in Power BI.
5. Structured telemetry in Azure Monitor tracks every step, with Kusto Query Language templates provided for audits and debugging.

Technology stack

Impact

• Volumes: Several million chats per year
• Revenue: +142% versus forecast as of March 2025.
• Conversion: +31% purchase conversion rate
• User satisfaction: AI Conversation CSAT over 4.0
• Efficiency: Human transfers down -46%; touchless product releases due to real-time detailed context of public pages

Try Store Assistant for yourself

In the United States, visit https://www.microsoft.com/en-us and click on the ‘Need help?’ widget at the bottom right.

Conclusion

Microsoft Store Assistant demonstrates how Semantic Kernel, Azure Open AI, AI Foundry and the broader Azure AI platform can transform a site without chat or a legacy scripted help bot into a dynamic assistant that understands an entire enterprise catalog, keeps customers safe, and improves continuously. As the industry quickly evolves and AI Foundry, Azure OpenAI, and Semantic Kernel lead in innovation, assistants like Store Assistant will only improve and increase in value and intelligence for customers.