Ground truth<\/strong> refers to a set of accurate, verified answers that serve as the benchmark against which an AI system\u2019s outputs are evaluated. It\u2019s the gold standard \u2014 the data you use to test whether the system is behaving as expected. In practice, ground truths are carefully curated question-and-answer pairs that reflect what users should<\/em> receive when they ask a particular question based on the system’s underlying data sources.<\/p>\n For example, if a user asks, \u201cWhat are the most recent updates related to this item?\u201d, the ground truth would be a verified, accurate list of those updates pulled directly from the system of record. This response represents the correct answer the AI system is expected to return and serves as the benchmark against which its performance can be tested and evaluated.<\/p>\n During a recent customer engagement, our team developed a structured approach to curate high-quality ground truths. This process was essential not just for evaluating the AI assistant\u2019s performance, but also for building confidence among end users and stakeholders.<\/p>\n In this post, we\u2019ll walk through the key steps in our approach:<\/p>\n Let\u2019s dive into the details of how we created ground truths that truly reflect user intent and organizational knowledge.<\/p>\n Once we\u2019ve established access to the underlying data sources our AI system will rely on, the first step in building reliable ground truths is to gather a meaningful set of real user questions. While it\u2019s possible to generate hypothetical prompts using the available data, these often miss the nuances, priorities, and terminology that actual users bring to the table. To develop ground truths that reflect real-world needs, we must engage directly with our target end users.<\/p>\n To do this, we conducted a focused, interactive workshop with subject matter experts (SMEs) \u2014 the people who understand the business domain and will ultimately use or benefit from the AI system. These sessions are not only about collecting questions, but also about building shared understanding and trust.<\/p>\n We began the workshop by clearly explaining the purpose of the project: to develop an AI assistant that can answer user questions accurately and consistently by drawing on existing organizational data. We emphasized how their expertise and participation were crucial to shaping an assistant that would be genuinely helpful and aligned with their day-to-day challenges.<\/p>\n To help SMEs generate high-quality questions, we gave them a set of thought-starters in advance. These prompts were designed to spark ideas and encourage them to think in terms of real scenarios:<\/p>\n During the session, we used a collaborative Mural board<\/a> \u2014 a virtual whiteboard with pre-labeled sections for different categories of questions. (If you have a Mural account, you can view our template here<\/a>.) Participants added their questions using digital sticky notes, and the format encouraged conversation, cross-pollination of ideas, and deeper exploration of user needs. Importantly, all questions were considered valid and valuable, no matter how specific, broad, or exploratory<\/strong>.<\/p>\n The workshop yielded dozens of user-submitted questions, many of which built on each other as SMEs discussed and refined their inputs in real time. In the second half of the session, we worked together to organize and tag the questions. We applied a color-coding system to indicate which data sources or systems would be required to answer each one. This step later enabled us to map questions to the appropriate context and data during the ground truth curation phase.<\/p>\n This collaborative question collection process not only gave us a strong foundation of realistic prompts, but also fostered user buy-in and deepened our understanding of the customer and their business \u2014 ensuring that the questions our AI assistant was being tested against were directly relevant to the people it was designed to serve.<\/p>\n Once we had a rich set of real user questions, the next step was to connect those questions to the actual data that could be used to answer them. This is what we call contextualized data curation<\/strong> \u2014 the process of identifying and extracting the relevant records or facts that represent a \u201ccorrect\u201d response to each user question based on the available data<\/em>.<\/p>\n This step is critical because it transforms abstract questions into concrete, testable pairs of input and expected output, which is the essence of a ground truth. It ensures that each ground truth is rooted not only in what users want<\/em> to know, but also in what the underlying data can support.<\/p>\n To streamline the effort, we grouped similar or related user questions into small sets. This allowed our team to work in parallel, with different individuals or subgroups focused on curating data for different clusters of questions. We also partnered with data scientists from the customer\u2019s organization, which helped accelerate progress and ensured alignment with internal data knowledge.<\/p>\n We developed custom tooling to support this phase. These tools enabled curators to define the data queries needed to answer each question and associate those queries with the relevant context using a well-defined, repeatable method.<\/p>\n For each user question, we followed a structured three-step process:<\/p>\n Determine which databases, data lakes, or structured files contain the information needed to answer the question.<\/li>\n For each relevant source, identify the fields and values that should be used to retrieve precise results. This might include timestamps, asset IDs, status fields, or other filters specific to the domain.<\/li>\n Construct a query that reliably returns the data needed for that specific question. These queries had to be complete and executable, returning interpretable results that could serve as the “correct” answer.<\/li>\n<\/ol>\n To give our queries real-world grounding and test their robustness, we introduced execution contexts<\/strong> \u2014 specific scenarios or entities that the query should operate on. For example, a query might be applied to a particular customer account, project, or product instance, depending on the domain.<\/p>\n We intentionally selected a mix of contexts to reflect different types of outcomes:<\/p>\n By including this range of contexts, we ensured that the curated ground truths represented a realistic cross-section of how users might interact with the system \u2014 from everyday questions to more difficult or uncommon scenarios.<\/p>\n This curated set of data-backed answers, anchored in well-defined contexts, became the foundation for the next step: automating the generation of structured ground truth files that could be validated and refined through SME review.<\/p>\n To operationalize the data curation process, we developed a utility that automates the generation of ground truth records at scale. For each set of user questions, we created a structured This input is fed into our Create Ground Truths Utility<\/strong>, which systematically runs each database query across all defined contexts for every user question. The utility then captures the resulting data records and packages them into a structured output file.<\/p>\n Each output These elements together form a ground truth entry<\/strong> \u2014 a verified, context-specific answer to a user question.<\/p>\n To further enrich the dataset, the utility also applies automated tagging<\/strong> to each ground truth. For example:<\/p>\n In addition to the primary This tooling helped ensure consistency and reproducibility while dramatically accelerating the process of building a robust, diverse set of ground truths.<\/p>\n Once we’ve curated contextualized data for each user question, the next critical step is validation<\/strong> \u2014 ensuring that the curated answers truly reflect what a domain expert would consider correct. After all, even a well-structured query can produce misleading or incomplete results if the intent of the question was misunderstood or if the wrong data properties were used.<\/p>\n To ensure the quality and credibility<\/strong> of our ground truths, we built a review loop centered on subject matter expert (SME) feedback<\/strong>. These experts are best positioned to assess whether the question was interpreted accurately and whether the returned data constitutes a valid and useful answer.<\/p>\n To streamline this feedback process, we created a utility that programmatically extracts the latest curated ground truths and converts them into a human-readable review format. Specifically, it generates an Each spreadsheet file includes:<\/p>\n Here is a generic example of an Excel spreadsheet showing contextualized ground truths for a fictionalized manufacturing plant that uses SAP terms and values:<\/p>\n SMEs can review each entry, provide feedback on interpretation, flag any issues, and suggest adjustments. This feedback loop is intentionally iterative<\/strong>. Once feedback is collected, the ground truth input files can be updated, and the utility can regenerate revised versions of each spreadsheet for further review.<\/p>\n Because the entire process \u2014 from query execution to spreadsheet generation \u2014 is automated, we can rapidly repeat the cycle of curation, validation, and refinement as needed.<\/p>\n This collaborative loop with SMEs ensures that the final ground truth dataset is not only technically sound, but also aligned with domain expertise and user expectations<\/strong> \u2014 a foundational step in delivering trusted AI systems.<\/p>\n High-quality ground truth data is the foundation of any reliable and trustworthy AI system. It ensures that models are not only trained on accurate examples but also evaluated against realistic, domain-relevant expectations.<\/p>\n By following a structured and collaborative process that collects authentic user questions, grounds those questions in real data, and validates the results with subject matter experts, teams can build ground truth datasets that reflect the complexity and nuance of real-world use cases.<\/p>\n This iterative approach doesn\u2019t just improve model performance. It strengthens alignment between technical teams and end users, ensures transparency in how AI systems are evaluated, and ultimately fosters trust in the solutions being delivered.<\/p>\n Investing in the ground truth process is more than a technical necessity \u2014 it\u2019s a strategic step toward building AI systems that are truly useful, dependable, and aligned with business goals.<\/p>\n We are grateful for the data science insights provided by Amatullah Badshah<\/a>, Farhan Baluch<\/a>, and Bryan Ostdiek<\/a> and the data expertise shared by Jose Perales<\/a>. We also appreciate Bindu Chinnasamy<\/a>, John Hauppa<\/a>, Cameron Taylor<\/a>, and Kanishk Tantia<\/a> for their valuable contributions to this project.<\/p>\n\n
![\"A](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/08\/ground-truth-process.png\")
<\/p>\n1. Collecting User Questions<\/h2>\n
\n
2. Contextualizing Data Curation<\/h2>\n
\n
\n
2.1 – Operations: Contextualized Data Curation Utility<\/h3>\n
JSON<\/code> input file. This file contains:<\/p>\n\n
JSON<\/code> file maps:<\/p>\n\n
\n
JSON<\/code> output, the utility generates JSONL<\/code> (JSON Lines) versions of the data. This format is especially useful for machine learning workflows, as it\u2019s compatible with platforms like Azure Machine Learning<\/a> (AML), where it can be used for testing, experimentation, and model evaluation.<\/p>\n![\"A](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/08\/create-ground-truths-utility-scaled.png\")
<\/p>\n3. Subject Matter Expert Validation<\/h2>\n
XLSX<\/code> (Excel) document containing worksheets for the latest batch of ground truth sets that need review.<\/p>\n\n
\n
![\"A](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/08\/validation-spreadsheet.png\")
<\/p>\n![\"A](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/08\/ground-truth-validation-process-scaled.png\")
<\/p>\nConclusion<\/h2>\n
Acknowledgments<\/h2>\n