2. For advanced search use case requiring ground truths and more precise measurement of retrieval quality, optimize your Search parameters using golden metrics such as XDCG and Max Relevance: Document Retrieval <\/a>evaluator<\/p>\n Agents can be powerful productivity enablers, they can increasingly understand business context and plan, make decisions, execute actions, or interact with human stakeholders or other agents to create more complex workflows for business needs. For example, RAG agents can use enterprise documents to ground their responses for relevance. However, the problems of agents as a black-box present significant challenges to developers building and observing agents. Developers need tools to assess quality and safety aspects of agent’s workflows.<\/p>\n [Demo] How to debug your RAG agent in Microsoft Foundry – framing agent observability<\/a><\/p>\n This post will focus on the best practices in RAG quality in agent workflows.<\/p>\n For complex queries that is a common use for RAG agents, we know that agentic RAG<\/a> is better than traditional RAG in principle and in practice. You can now use agentic retrieval API<\/a> (aka Knowledge Agent) shown to improve Up to 40% better relevance for complex queries with new agentic retrieval engine<\/a> than traditional RAG.<\/p>\n At a high-level, a retrieval-augmented generation (RAG) system tries to generate the most relevant answer consistent with grounding documents in response to a user’s query. At a high level, a user’s query triggers a search retrieval in the corpus of grounding documents to provide grounding context for the AI model to generate a response. It’s important to evaluate the following aspects using our evaluators (aka the “RAG triad” metrics):<\/p>\n In this best practice, we will focus on the evaluating the end-to-end response of Knowledge Agent using Groundedness and Relevance evaluators.<\/p>\n What can Knowledge Agent do? Knowledge Agent is an advanced agentic retrieval pipeline designed to extract grounding information from your knowledge sources. Given the conversation history and retrieval parameters, the agent:<\/p>\n Document retrieval quality is a common bottleneck in RAG Search. One best practice is to optimize your RAG search parameters according to your enterprise data. For advanced users who can curate ground truth for document retrieval results – query relevance labels (“qrels” for short), it is an important scenario to “sweep” and thus optimize the parameters by evaluating the document retrieval quality, using golden metrics such as XDCG and Max Relevance.<\/p>\nFraming Agent Observability<\/h3>\n
Best Practice 1: Evaluate your RAG end to end<\/h3>\n
![\"RAG](\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/08\/RAG-triads-300x169.png\")
<\/a><\/p>\n\n
\n
from azure.search.documents.agent import KnowledgeAgentRetrievalClient\r\nfrom azure.search.documents.agent.models import (\r\n\u00a0 \u00a0 KnowledgeAgentRetrievalRequest, KnowledgeAgentMessage, KnowledgeAgentMessageTextContent, SearchIndexKnowledgeSourceParams\r\n)\r\n\r\n# test a knowledge-intense query or two\r\n\r\nagent_client = KnowledgeAgentRetrievalClient(endpoint=endpoint, agent_name=agent_name, credential=credential)\r\nquery_1 = \"\"\"\r\n\u00a0 \u00a0 Why do suburban belts display larger December brightening than urban cores even though absolute light levels are higher downtown?\r\n\u00a0 \u00a0 Why is the Phoenix nighttime street grid is so sharply visible from space, whereas large stretches of the interstate between midwestern cities remain comparatively dim?\r\n\u00a0 \u00a0 \"\"\"\r\nmessages.append({\r\n\u00a0 \u00a0 \"role\": \"user\",\r\n\u00a0 \u00a0 \"content\": query_1\r\n})\r\nretrieval_request= KnowledgeAgentRetrievalRequest(\r\n\u00a0 \u00a0 messages=[\r\n\u00a0 \u00a0 \u00a0 \u00a0 KnowledgeAgentMessage(\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 role=m[\"role\"],\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 content=[KnowledgeAgentMessageTextContent(text=m[\"content\"])]\r\n\u00a0 \u00a0 \u00a0 \u00a0 ) for m in messages if m[\"role\"] != \"system\"\r\n\u00a0 \u00a0 ],\r\n\u00a0 \u00a0 knowledge_source_params=[\r\n\u00a0 \u00a0 \u00a0 \u00a0 SearchIndexKnowledgeSourceParams(\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 knowledge_source_name=knowledge_source_name,\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 kind=\"searchIndex\"\r\n\u00a0 \u00a0 \u00a0 \u00a0 )\r\n\u00a0 \u00a0 ]\r\n)\r\nresult = agent_client.retrieve(retrieval_request=retrieval_request, api_version=api_version)\r\nprint(\"Retrieve successful\")\r\n\r\n\r\nimport json\r\n# Build simple string values for response_content, activity_content, references_content\r\n# Responses -> concatenate all text\/value fields from all response contents\r\nresponse_parts = []\r\nif getattr(result, \"response\", None):\r\n\u00a0 \u00a0 for resp in result.response:\r\n\u00a0 \u00a0 \u00a0 \u00a0 for content in getattr(resp, \"content\", []):\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 text = getattr(content, \"text\", None) or getattr(content, \"value\", None) or str(content)\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 response_parts.append(text)\r\nresponse_content = \"\\n\\n\".join(response_parts) if response_parts else \"No response found on 'result'\"\r\nresponse_contents.append(response_content)\r\n# Print the three string values\r\nprint(\"response_content:\\n\", response_content, \"\\n\")<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n\n
## Run Evaluation with Microsoft Foundry\r\nai_foundry_project_endpoint = os.environ.get(\"AI_FOUNDRY_PROJECT_ENDPOINT\")\r\nimport os, json\r\nfrom azure.ai.evaluation import (\r\n\u00a0 \u00a0 AzureOpenAIModelConfiguration,\r\n\u00a0 \u00a0 GroundednessEvaluator,\r\n\u00a0 \u00a0 RelevanceEvaluator,\r\n\u00a0 \u00a0 RetrievalEvaluator,\r\n\u00a0 \u00a0 evaluate,\r\n)\r\nevaluation_data = []\r\nfor q, r, g in zip([query_1, query_2], references_contents, response_contents):\r\n\u00a0 \u00a0 evaluation_data.append({\r\n\u00a0 \u00a0 \u00a0 \u00a0 \"query\": q,\r\n\u00a0 \u00a0 \u00a0 \u00a0 \"response\": g,\r\n\u00a0 \u00a0 \u00a0 \u00a0 \"context\": r,\r\n\u00a0 \u00a0 })\r\nfilename = \"evaluation_data.jsonl\"\r\nwith open(filename, \"w\") as f:\r\n\u00a0 \u00a0 for item in evaluation_data:\r\n\u00a0 \u00a0 \u00a0 \u00a0 f.write(json.dumps(item) + \"\\n\")\r\nmodel_config = AzureOpenAIModelConfiguration(\r\n\u00a0 \u00a0 azure_endpoint=azure_openai_endpoint,\r\n\u00a0 \u00a0 api_version=azure_openai_api_version,\r\n\u00a0 \u00a0 azure_deployment=azure_openai_gpt_model\r\n)\r\n# RAG triad metrics\r\ngroundedness = GroundednessEvaluator(model_config=model_config)\r\nrelevance = RelevanceEvaluator(model_config=model_config)\r\nresult = evaluate(\r\n\u00a0 \u00a0 data=filename,\r\n\u00a0 \u00a0 evaluators={\r\n\u00a0 \u00a0 \u00a0 \u00a0 \"groundedness\": groundedness,\r\n\u00a0 \u00a0 \u00a0 \u00a0 \"relevance\": relevance,\r\n\u00a0 \u00a0 },\r\n\u00a0 \u00a0 azure_ai_project=ai_foundry_project_endpoint,\r\n)\r\nprint(\"Evaluation complete.\")\r\nstudio_url = result.get(\"studio_url\")\r\nif studio_url:\r\n\u00a0 \u00a0 print(\"AI Foundry Studio URL:\", studio_url)<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\nBest Practice 2: Optimize your RAG search parameters<\/h3>\n
![\"Doc](\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Doc-retrieval-list-view-300x198.png\")
<\/p>\n![\"Doc](\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Doc-retrieval-compare-view-300x87.png\")
<\/p>\n![\"Doc](\"https:\/\/devblogs.microsoft.com\/foundry\/wp-content\/uploads\/sites\/89\/2025\/05\/Doc-retrieval-best-parameter-xdcg-chart-300x150.png\")
<\/p>\n