{"id":530,"date":"2025-05-19T09:00:29","date_gmt":"2025-05-19T16:00:29","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/foundry\/?p=530"},"modified":"2025-05-21T10:38:24","modified_gmt":"2025-05-21T17:38:24","slug":"achieve-end-to-end-observability-in-azure-ai-foundry","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/foundry\/achieve-end-to-end-observability-in-azure-ai-foundry\/","title":{"rendered":"Achieve End-to-End Observability in Azure AI Foundry"},"content":{"rendered":"<p><span class=\"TextRun SCXW154155407 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW154155407 BCX8\">Today, <\/span><span class=\"NormalTextRun SCXW154155407 BCX8\">we\u2019re<\/span><span class=\"NormalTextRun SCXW154155407 BCX8\"> thrilled to launch the <\/span><\/span><span class=\"TextRun SCXW154155407 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW154155407 BCX8\">public preview of Azure AI Foundry Observability<\/span><\/span><span class=\"TextRun SCXW154155407 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW154155407 BCX8\">, the first unified solution for <\/span><\/span><span class=\"TextRun SCXW154155407 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW154155407 BCX8\">governance, evaluation, tracing, and monitoring<\/span><\/span><span class=\"TextRun SCXW154155407 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"><span class=\"NormalTextRun SCXW154155407 BCX8\"> \u2014 all built into your AI de<\/span><span class=\"NormalTextRun SCXW154155407 BCX8\">velopment<\/span><span class=\"NormalTextRun SCXW154155407 BCX8\"> loop. From model selection to real-time debugging, <\/span><span class=\"NormalTextRun SCXW154155407 BCX8\">our<\/span><span class=\"NormalTextRun SCXW154155407 BCX8\"> observability <\/span><span class=\"NormalTextRun SCXW154155407 BCX8\">capabilities<\/span><span class=\"NormalTextRun SCXW154155407 BCX8\"> empower teams to ship production-grade AI with confidence and speed.<\/span><\/span><span class=\"EOP Selected SCXW154155407 BCX8\" data-ccp-props=\"{&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><figure id=\"attachment_581\" aria-labelledby=\"figcaption_attachment_581\" class=\"wp-caption aligncenter\" ><a href=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Picture1.png\"><img decoding=\"async\" class=\"size-full wp-image-581\" src=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Picture1.png\" alt=\"\" width=\"489\" height=\"312\" srcset=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Picture1.png 489w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Picture1-300x191.png 300w\" sizes=\"(max-width: 489px) 100vw, 489px\" \/><\/a><figcaption id=\"figcaption_attachment_581\" class=\"wp-caption-text\">Figure 1: Observability aligned with the end-to-end AI application development workflow.<\/figcaption><\/figure><\/p>\n<h2 aria-level=\"1\"><span data-contrast=\"none\">See Everything, From Prototype to Production<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:360,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">Foundry Observability brings continuous visibility across your entire AI application lifecycle. Whether you are prototyping, actively developing with CI\/CD pipelines \u2014 we provide the capabilities you need to assess, monitor, scale and optimize your AI agents.<\/span><\/p>\n<p><iframe title=\"YouTube video player\" src=\"\/\/www.youtube.com\/embed\/KAXuwfzXk48?si=v_YzUIW-nLfh6BMA\" width=\"560\" height=\"315\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<h3 aria-level=\"2\"><span data-contrast=\"none\">Kickstart Development<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:281,&quot;335559739&quot;:281}\">\u00a0<\/span><\/h3>\n<h4 aria-level=\"3\"><span data-contrast=\"none\">AI Governance, Streamlined<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h4>\n<p><span data-contrast=\"none\">We\u2019re bringing responsible AI front and center with new governance integrations in Azure AI Foundry. Now you can connect with <\/span><span data-contrast=\"auto\">Microsoft Purview<\/span><span data-contrast=\"none\"> Compliance Manager,\u00a0<\/span> <a href=\"https:\/\/www.credo.ai\/?utm_term=credo%20ai&amp;utm_campaign=Brand+%7C+3\/23&amp;utm_source=adwords&amp;utm_medium=ppc&amp;hsa_acc=9234903900&amp;hsa_cam=19907119827&amp;hsa_grp=146052758765&amp;hsa_ad=652950490050&amp;hsa_src=g&amp;hsa_tgt=kwd-1515820219902&amp;hsa_kw=credo%20ai&amp;hsa_mt=p&amp;hsa_net=adwords&amp;hsa_ver=3&amp;gad_source=1&amp;gad_campaignid=19907119827&amp;gbraid=0AAAAAoUale4A6LHdtfsc1O6cm_-ZgE-8M&amp;gclid=Cj0KCQjwlYHBBhD9ARIsALRu09pccgusCYN5VyMwi5wjQIn7qgL6_sR111YNRdoYif_Eb2-FOUpi1BUaApdkEALw_wcB\"><span data-contrast=\"none\">Credo AI<\/span><\/a><span data-contrast=\"auto\"> and <\/span><a href=\"https:\/\/www.saidot.ai\/\"><span data-contrast=\"none\">Saidot<\/span><\/a><span data-contrast=\"auto\">,<\/span><span data-contrast=\"none\"> to define evaluation plans aligned with frameworks like the EU AI Act \u2014 and run them directly via the Azure AI Evaluation SDK. No guesswork, just streamlined, audit-ready governance built into your dev workflow.<\/span><span data-ccp-props=\"{&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">To dive deeper into how these integrations work in practice, check out our AI governance <\/span><a href=\"https:\/\/aka.ms\/AIgovernanceBuild2025\"><span data-contrast=\"none\">blog post<\/span><\/a><span data-contrast=\"auto\">.<\/span><span data-ccp-props=\"{&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<h4 aria-level=\"3\"><span data-contrast=\"none\">Leaderboards That Lead the Way<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h4>\n<p><span data-contrast=\"none\">Choosing the right model just got easier. Azure AI Foundry\u2019s <\/span><a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/ai-foundry\/how-to\/benchmark-model-in-catalog\"><span data-contrast=\"none\">new leaderboards<\/span><\/a><span data-contrast=\"none\"> let you compare foundation models by quality, cost, and performance \u2014 all backed by industry benchmarks. Visualize trade-offs, explore scenario-based rankings, and dive into quality, performance, and cost metrics to enhance your \u201cmodel shopping\u201d experience. Fast, confident model selection starts here.<\/span><span data-ccp-props=\"{&quot;335559738&quot;:120,&quot;335559739&quot;:60}\">\u00a0<\/span><\/p>\n<p><figure id=\"attachment_582\" aria-labelledby=\"figcaption_attachment_582\" class=\"wp-caption aligncenter\" ><a href=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Picture2.png\"><img decoding=\"async\" class=\"size-full wp-image-582\" src=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Picture2.png\" alt=\"\" width=\"1430\" height=\"704\" srcset=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Picture2.png 1430w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Picture2-300x148.png 300w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Picture2-1024x504.png 1024w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Picture2-768x378.png 768w\" sizes=\"(max-width: 1430px) 100vw, 1430px\" \/><\/a><figcaption id=\"figcaption_attachment_582\" class=\"wp-caption-text\">Figure 2: Model leaderboards UI in the Azure AI Foundry portal.<\/figcaption><\/figure><\/p>\n<h4 aria-level=\"3\"><span data-contrast=\"none\">Evaluate and Debug with Traces in the Agents Playground See Inside Your Agent<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h4>\n<p><span data-contrast=\"none\">The <\/span><a href=\"https:\/\/aka.ms\/agents-playground\"><span data-contrast=\"none\">Agents Playground<\/span><\/a><span data-contrast=\"none\"> now comes with built-in evaluation and tracing \u2014 so you can test, debug, and improve your agents in one place. Quality checks run by default, safety checks are just a toggle away, and every result is trace-linked for full visibility into tool calls, inputs, outputs, and metrics.<\/span><span data-ccp-props=\"{&quot;335559738&quot;:120,&quot;335559739&quot;:60}\">\u00a0<\/span><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Agents-Playground.jpg\"><img decoding=\"async\" class=\"aligncenter wp-image-589 size-full\" src=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Agents-Playground.jpg\" alt=\"\" width=\"539\" height=\"441\" srcset=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Agents-Playground.jpg 539w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Agents-Playground-300x245.jpg 300w\" sizes=\"(max-width: 539px) 100vw, 539px\" \/><\/a><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/image-32-1.png\"><img decoding=\"async\" class=\"aligncenter wp-image-666 size-large\" src=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/image-32-1-1024x430.png\" alt=\"image 32 image\" width=\"1024\" height=\"430\" srcset=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/image-32-1-1024x430.png 1024w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/image-32-1-300x126.png 300w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/image-32-1-768x323.png 768w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/image-32-1-1536x646.png 1536w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/image-32-1-2048x861.png 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<h3 aria-level=\"2\"><span data-contrast=\"none\">Phase 2: Transition to Code<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h3>\n<h4 aria-level=\"3\"><span data-contrast=\"none\">Evaluate What Matters<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h4>\n<p><span data-contrast=\"auto\">We\u2019ve supercharged agent evaluation in Azure AI Foundry. You can now directly assess agent thread messages using built-in metrics like:<\/span><\/p>\n<table style=\"border-collapse: collapse; width: 100%;\">\n<tbody>\n<tr>\n<td style=\"width: 50%;\">Intent Resolution<\/td>\n<td style=\"width: 50%;\">Measures how accurately the agent identifies and addresses user intentions.<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 50%;\">Task Adherence<\/td>\n<td style=\"width: 50%;\">Measures how well the agent follows through on identified tasks.<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 50%;\">Tool Call Accuracy<\/td>\n<td style=\"width: 50%;\">Measures how well the agent selects and calls the correct tools.<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 50%;\">Response Completeness<\/td>\n<td style=\"width: 50%;\">Measures to what extent the response is complete (not missing critical information) with respect to the ground truth.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><span data-contrast=\"auto\">No extra parsing needed \u2014 just plug in and go, even if you&#8217;re building outside Azure AI Agent Service.<\/span><span data-ccp-props=\"{&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">And with our new integrations with <\/span><a href=\"http:\/\/learn.microsoft.com\/en-us\/azure\/ai-foundry\/concepts\/evaluation-evaluators\/azure-openai-graders\"><b><span data-contrast=\"none\">Azure OpenAI Graders<\/span><\/b><\/a><span data-contrast=\"auto\">, you get even more precision:<\/span><span data-ccp-props=\"{&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"28\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\">Label Grader<\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"28\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\">String Checker<\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"28\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\">Text Similarity<\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"28\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\">Custom General Grader<span data-ccp-props=\"{&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><span data-contrast=\"auto\">Together, these tools give you a full-spectrum view of agent quality and safety \u2014 from prototype to production.<\/span><span data-ccp-props=\"{&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/image-33-1.png\"><img decoding=\"async\" class=\"aligncenter wp-image-667 size-large\" src=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/image-33-1-1024x455.png\" alt=\"image 33 image\" width=\"1024\" height=\"455\" srcset=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/image-33-1-1024x455.png 1024w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/image-33-1-300x133.png 300w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/image-33-1-768x341.png 768w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/image-33-1-1536x683.png 1536w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/image-33-1-2048x910.png 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/a><\/p>\n<h4 aria-level=\"3\"><span data-contrast=\"none\">Scan for Vulnerabilities with AI Red Teaming Agent<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h4>\n<p><span data-contrast=\"auto\">Meet the <a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/ai-foundry\/how-to\/develop\/run-scans-ai-red-teaming-agent\" target=\"_blank\" rel=\"noopener\">Azure AI Foundry AI Red Teaming Agent<\/a> \u2014 your built-in defense against unsafe AI. Powered by Microsoft\u2019s open-source PyRIT, it simulates adversarial attacks to uncover vulnerabilities before you ship.<\/span><span data-ccp-props=\"{&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"27\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Scan for content safety risks automatically<\/span><span data-ccp-props=\"{&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"27\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Measure exposure with metrics like Attack Success Rate (ASR)<\/span><span data-ccp-props=\"{&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"27\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Generate detailed readiness reports<\/span><span data-ccp-props=\"{&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><span data-contrast=\"auto\">No specialized expertise required. Just plug it into your workflow and build with confidence.<\/span><span data-ccp-props=\"{&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">For a deeper dive into the capabilities and implementation details of the AI Red Teaming Agent, check out our dedicated <\/span><a href=\"https:\/\/devblogs.microsoft.com\/foundry\/ai-red-teaming-agent-preview\/\"><span data-contrast=\"none\">AI Red Teaming blog post.<\/span><\/a><span data-ccp-props=\"{&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/metric-dashboard-red-team-1.png\"><img decoding=\"async\" class=\"aligncenter wp-image-668 size-large\" src=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/metric-dashboard-red-team-1-1024x187.png\" alt=\"metric dashboard red team image\" width=\"1024\" height=\"187\" srcset=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/metric-dashboard-red-team-1-1024x187.png 1024w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/metric-dashboard-red-team-1-300x55.png 300w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/metric-dashboard-red-team-1-768x140.png 768w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/metric-dashboard-red-team-1-1536x280.png 1536w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/metric-dashboard-red-team-1-2048x373.png 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/a> <a href=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/detailed-metrics-results-1.png\"><img decoding=\"async\" class=\"aligncenter wp-image-665 size-large\" src=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/detailed-metrics-results-1-1024x526.png\" alt=\"detailed metrics results image\" width=\"1024\" height=\"526\" srcset=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/detailed-metrics-results-1-1024x526.png 1024w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/detailed-metrics-results-1-300x154.png 300w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/detailed-metrics-results-1-768x395.png 768w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/detailed-metrics-results-1-1536x789.png 1536w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/detailed-metrics-results-1-2048x1052.png 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/a><\/p>\n<h4 aria-level=\"3\"><span data-contrast=\"none\">CI\/CD-Ready from Day One<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h4>\n<p><span data-contrast=\"auto\">Azure AI Foundry now plugs straight into your CI\/CD workflows. With our <\/span><a href=\"https:\/\/aka.ms\/Eval-GitHub-Action\"><span data-contrast=\"none\">GitHub Action<\/span><\/a><span data-contrast=\"auto\"> and <\/span><a href=\"https:\/\/aka.ms\/Eval-ADO-Workflow\"><span data-contrast=\"none\">Azure DevOps Extension<\/span><\/a><span data-contrast=\"auto\">, you can:<\/span><span data-ccp-props=\"{&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"26\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Auto-evaluate agents on every commit<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"26\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Compare versions with built-in quality, performance, and safety metrics<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"26\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Get confidence intervals and significance tests to back your decisions<\/span><span data-ccp-props=\"{&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><span data-contrast=\"auto\">It\u2019s continuous evaluation, made continuous.<\/span><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Evals-GitHub-Action-05192025.gif\" \/><\/p>\n<p>&nbsp;<\/p>\n<h3 aria-level=\"2\"><span data-contrast=\"none\">Phase #3: Operate in Production<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h3>\n<h4 aria-level=\"3\"><span data-contrast=\"none\"> Monitor in Production, Effortlessly<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h4>\n<p><span data-contrast=\"auto\">Once your agent is live, Azure AI Foundry keeps watch and enables <\/span><a href=\"https:\/\/aka.ms\/monitoring-apps\"><span data-contrast=\"none\">continuous monitoring<\/span><\/a><span data-contrast=\"auto\">. A unified dashboard tracks performance, quality, safety, and resource usage \u2014 all in real time.<\/span><span data-ccp-props=\"{&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"25\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Run <\/span><a href=\"https:\/\/aka.ms\/Continuous-Eval-Agents\"><span data-contrast=\"none\">continuous evaluations<\/span><\/a><span data-contrast=\"auto\"> on live traffic (e.g., 10 per hour)<\/span><span data-ccp-props=\"{&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"25\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Set alerts in Azure Monitor to catch drift or regressions<\/span><span data-ccp-props=\"{&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"25\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Link directly to Azure Monitor Application Insights for full-stack visibility<\/span><span data-ccp-props=\"{&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><span data-contrast=\"auto\">From metrics to traces, you\u2019ve got everything you need to stay ahead of issues.<\/span><span data-ccp-props=\"{&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Monitoring-v2-1.gif\"><img decoding=\"async\" class=\"aligncenter wp-image-704 size-large\" src=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Monitoring-v2-1-1024x629.gif\" alt=\"Monitoring v2 image\" width=\"1024\" height=\"629\" srcset=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Monitoring-v2-1-1024x629.gif 1024w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Monitoring-v2-1-300x184.gif 300w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Monitoring-v2-1-768x471.gif 768w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Monitoring-v2-1-1536x943.gif 1536w, https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Monitoring-v2-1-2048x1257.gif 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/a><\/p>\n<p><span data-contrast=\"auto\">This unified dashboard above is powered by Azure Monitor Application Insights and Azure Workbooks, which allows you to monitor app performance in the broader context of your infrastructure. You can navigate seamlessly from Foundry Observability to Azure Monitor for advanced monitoring capabilities, such as the ability to customize monitoring dashboards and set up alerts for advanced diagnostics and incident response.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<h4 aria-level=\"3\"><span data-contrast=\"none\">Trace Every Evaluation<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h4>\n<p><span data-contrast=\"auto\"> With <\/span><a href=\"https:\/\/aka.ms\/tracing-app\"><span data-contrast=\"none\">tracing<\/span><\/a><span data-contrast=\"auto\"> enabled, every evaluation result is mapped to a trace \u2014 giving you full visibility into your agent\u2019s execution flow. From LLM inference to tool calls, inputs, outputs, and metrics, you can debug regressions (like groundedness drops) with precision and speed.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Tracing-05162025.gif\" \/><\/p>\n<p>&nbsp;<\/p>\n<h4 aria-level=\"3\"><span data-contrast=\"none\">Pricing<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:160,&quot;335559739&quot;:80}\">\u00a0<\/span><\/h4>\n<p>AI-assisted evaluations and monitoring, risk and safety,\u202fincur charges of:<\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"30\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b>$20<\/b>\/1M input tokens<\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"30\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;multilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b>$60<\/b>\/1M output tokens<\/li>\n<\/ul>\n<p>For all other evaluation metrics (NLP metrics), see\u202f<a href=\"https:\/\/azure.microsoft.com\/en-us\/pricing\/details\/machine-learning\/\">compute costs<\/a>.<\/p>\n<p><i><span data-contrast=\"auto\">Prices are estimates only and are not intended as actual price quotes. Actual pricing may vary depending on the type of agreement entered with Microsoft, date of purchase, and the currency exchange rate. Prices are calculated based on US dollars and converted using London closing spot rates that are captured in the two business days prior to the last business day of the previous month end. If the two business days prior to the end of the month fall on a bank holiday in major markets, the rate setting day is generally the day immediately preceding the two business days. This rate applies to all transactions during the upcoming month. Sign in to the\u202f<\/span><\/i><a href=\"https:\/\/azure.microsoft.com\/en-us\/pricing\/calculator\/\"><i><span data-contrast=\"none\">Azure pricing calculator<\/span><\/i><\/a><i><span data-contrast=\"auto\">\u202fto see pricing based on your current program\/offer with Microsoft. Contact an\u202f<\/span><\/i><a href=\"https:\/\/azure.microsoft.com\/en-us\/contact\/pricing\/\"><i><span data-contrast=\"none\">Azure sales specialist<\/span><\/i><\/a><i><span data-contrast=\"auto\">\u202ffor more information on pricing or to request a price quote. See\u202f<\/span><\/i><a href=\"https:\/\/azure.microsoft.com\/en-us\/pricing\/\"><i><span data-contrast=\"none\">frequently asked questions<\/span><\/i><\/a><i><span data-contrast=\"auto\">\u202fabout Azure pricing.<\/span><\/i><\/p>\n<h3><span data-contrast=\"auto\">Additional Resources<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/h3>\n<ul>\n<li><a href=\"https:\/\/aka.ms\/agents-playground\"><span data-contrast=\"auto\">Agents playground evaluations\u00a0<\/span><\/a><\/li>\n<li><a href=\"https:\/\/aka.ms\/Eval-GitHub-Action\"><span data-contrast=\"auto\">How to run an evaluation in GitHub Action\u00a0<\/span><\/a><\/li>\n<li><span data-contrast=\"auto\"><a href=\"https:\/\/aka.ms\/Eval-ADO-Workflow\">Evaluation ADO Workflow Extension<\/a><\/span><\/li>\n<li><span data-contrast=\"auto\"><a href=\"https:\/\/aka.ms\/monitoring-apps\">Continuous monitoring<\/a><\/span><\/li>\n<li><a href=\"https:\/\/aka.ms\/Continuous-Eval-Agents\"><span data-contrast=\"auto\">Continuously evaluate your AI agents<\/span><\/a><\/li>\n<li><a href=\"https:\/\/aka.ms\/tracing-app\"><span data-contrast=\"auto\">Tracing<\/span><\/a><\/li>\n<li><a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/ai-foundry\/concepts\/ai-red-teaming-agent\"><span data-contrast=\"auto\">AI Red Teaming Agent<\/span><\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Announcing public preview of Azure AI Foundry Observability, a unified solution for governance, evaluation, tracing, and monitoring in AI development. This innovative tool empowers teams to confidently ship production-grade AI by providing continuous visibility across the entire AI application lifecycle, from model selection to real-time debugging.<\/p>\n","protected":false},"author":190399,"featured_media":598,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[3,34,2],"class_list":["post-530","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-microsoft-foundry","tag-ai-development","tag-microsoft-build","tag-microsoft-foundry"],"acf":[],"blog_post_summary":"<p>Announcing public preview of Azure AI Foundry Observability, a unified solution for governance, evaluation, tracing, and monitoring in AI development. This innovative tool empowers teams to confidently ship production-grade AI by providing continuous visibility across the entire AI application lifecycle, from model selection to real-time debugging.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/posts\/530","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/users\/190399"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/comments?post=530"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/posts\/530\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/media\/598"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/media?parent=530"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/categories?post=530"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/foundry\/wp-json\/wp\/v2\/tags?post=530"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}