{"id":25662,"date":"2026-05-08T13:51:57","date_gmt":"2026-05-08T20:51:57","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/microsoft365dev\/?p=25662"},"modified":"2026-05-11T09:40:19","modified_gmt":"2026-05-11T16:40:19","slug":"announcing-the-public-preview-of-the-microsoft-365-copilot-agent-evaluations-tool","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/microsoft365dev\/announcing-the-public-preview-of-the-microsoft-365-copilot-agent-evaluations-tool\/","title":{"rendered":"Announcing the public preview of the Microsoft 365 Copilot Agent Evaluations tool"},"content":{"rendered":"

Today we’re announcing the public preview of the <\/span>Microsoft 365 Copilot Agent Evaluations tool<\/span><\/a>. The Agent Evaluations CLI tool helps developers measure and improve the quality of agents they build for Microsoft 365 Copilot. The tool provides a command-line interface that sends prompts to a deployed agent, captures responses, and scores them with the help of Azure OpenAI LLM models. It produces structured reports developers can use in their inner loop of development and in CI\/CD pipelines.<\/span>\u00a0<\/span><\/p>\n

This release is a step toward making rigorous, repeatable evaluation a standard part of how developers build for Microsoft 365 Copilot, alongside the broader work happening across the platform, from Work IQ to agent creation with the Microsoft 365 Agents Toolkit.<\/span>\u00a0<\/span><\/p>\n

Why evaluations\u00a0matter now<\/span>\u00a0<\/span><\/h2>\n

Our mission is to enable Microsoft partners, ISVs, and enterprise developers to extend Microsoft 365 Copilot with custom agents, actions, and knowledge so Copilot can reason over any data and take action across any system.<\/span>\u00a0<\/span><\/p>\n

As agents move from demos into core business workflows, the bar for shipping rises with them. Customers expect agents that are accurate, grounded, and consistent across the breadth of real-world prompts they receive. Meeting that bar requires more than manual testing. It requires an evaluation framework that is objective, repeatable, and integrated into the developer workflow. The Agent Evaluations tool is designed to make that practical.<\/span>\u00a0<\/span><\/p>\n

What’s in the public preview<\/span>\u00a0<\/span><\/h2>\n

The public preview brings the full evaluation loop into a simple command-line workflow. The CLI is designed to fit naturally into the way Microsoft 365 developers already build agents. Developers can invoke the CLI to evaluate declarative agents right inside the Microsoft 365 Agents Toolkit.<\/span><\/p>\n