...Your task is to generate a correct PostgreSQL query to answer the user's [Question]. You may be given a [Hint] that will help you solve the question...\r\n\r\n[Hint]:\r\n- Income Classification (value_illustration):\r\n Illustrates the income brackets for household economic status.\r\n Ranges from 'Low Income' to 'Very High Income'. Null indicates undisclosed or irregular income.<\/code><\/pre>\n3. Agent reasoning<\/strong> – We captured interim messages and function calls in a\nlog file:<\/p>\n{\"type\": \"call\", \"function\": \"get_db_table_list\", \"arguments\": {}}\r\n{\"type\": \"return\", \"result\": \"amenities\\nhouseholds\\n...\"}\r\n{\"type\": \"call\", \"function\": \"get_tb_table_schema\", \"arguments\": {\"name\": \"households\"}}<\/code><\/pre>\n4. Predicted SQL<\/strong> – The final output given by the agent:<\/p>\nSELECT DISTINCT households.housenum\r\nFROM households\r\nJOIN properties ON properties.houselink = households.housenum\r\nJOIN amenities ON amenities.houseid = households.housenum\r\nWHERE TRIM(UPPER(households.locregion)) = 'GUAR\u00c1'\r\n AND TRIM(UPPER(properties.dwelling_specs->>'Dwelling_Class')) IN ('BRICKWORK HOUSE','APARTMENT','CONDOMINIUM')\r\n AND TRIM(UPPER(amenities.cablestatus)) IN ('AVAIL','AVAILABLE','YES')\r\nORDER BY households.housenum;<\/code><\/pre>\n5. Results<\/strong> – The predicted SQL is then executed against the database and\ncompared with the ground truth.<\/p>\n{\"housenum\": \"35\"}, {\"housenum\": \"234\"}, ...<\/code><\/pre>\nExperiments<\/h2>\n
We systematically evaluated what factors improve or hinder NL-to-SQL accuracy\nacross three approaches. The experiments revealed clear patterns in system\nperformance and identified both solvable technical challenges and fundamental\nlimitations.<\/p>\n
Experimental Journey<\/h3>\n
Our experiments followed an iterative approach focused on understanding how\nmetadata and validation mechanisms improve query generation. We tested each\nchange against our standardized evaluation dataset, allowing us to isolate the\nimpact of individual improvements. The results below show this progression.<\/p>\n
For clarity in the tables below: Metadata<\/strong> refers to technical schema context\n(table and column descriptions, types, relationships), and domain hints<\/strong> are\nbusiness domain instructions we inject into prompts or a knowledge store to\nexplain what fields mean and how metrics should be computed.<\/p>\nExperiment Overview<\/h3>\nGitHub Copilot CLI<\/h4>\n
\n\n\n| Model<\/th>\n | Experiment Description<\/th>\n | Accuracy<\/th>\n<\/tr>\n<\/thead>\n |
\n\n| Claude Sonnet 4.5<\/td>\n | Metadata only<\/td>\n | 66.70%<\/td>\n<\/tr>\n |
\n| Claude Sonnet 4.5<\/td>\n | Metadata and domain hints<\/td>\n | 80.77%<\/td>\n<\/tr>\n |
\n| Gemini 3.0 Pro Preview<\/td>\n | Metadata and domain hints<\/td>\n | 74.07%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nMicrosoft Agent Framework<\/h4>\n\n\n\n| Model<\/th>\n | Experiment Description<\/th>\n | Accuracy<\/th>\n<\/tr>\n<\/thead>\n | \n\n| GPT-5 Mini<\/td>\n | Metadata only<\/td>\n | 55.60%<\/td>\n<\/tr>\n | \n| GPT-5 Mini<\/td>\n | Metadata and domain hints<\/td>\n | 65.38%<\/td>\n<\/tr>\n | \n| GPT-5 Mini<\/td>\n | Metadata and additional clarification agent tool providing domain knowledge<\/td>\n | 69.23%<\/td>\n<\/tr>\n | \n| GPT-5 Mini<\/td>\n | Metadata and domain hints with revisions to agent instructions<\/td>\n | 76.92%<\/td>\n<\/tr>\n | \n| GPT-5 Mini<\/td>\n | Ablation study – removing ability to query data<\/td>\n | 38.46%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nAzure Databricks AI\/BI Genie<\/h4>\n\n\n\n| Experiment Description<\/th>\n | Accuracy<\/th>\n<\/tr>\n<\/thead>\n | \n\n| Empty Spaces – no metadata or knowledge store<\/td>\n | 9.50%<\/td>\n<\/tr>\n | \n| Added column descriptions plus system instructions with domain knowledge<\/td>\n | 69.23%<\/td>\n<\/tr>\n | \n| Feedback mechanism – storing SQL patterns and joins from 3 feedback iterations<\/td>\n | 88.50%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nFindings<\/h2>\nSchema Knowledge + Live Validation<\/h3>\nField-level metadata (technical schema details like column types and\nconstraints) and documentation (semantic understanding of what fields mean) are\nboth critical to successful query generation. Runtime database access allows the\nagent to test and validate its SQL before producing the final result.<\/p>\n Giving the agent better schema\/context lifted accuracy across all systems, and\nletting it actually run queries to sanity-check its SQL was essential. When we\nremoved runtime querying, accuracy collapsed to 38.46%, so metadata alone was\nnot enough\u2014execution feedback mattered.<\/p>\n The gain from having metadata available was substantial: AI\/BI Genie improved\nfrom 9.50% to 69.23%; Copilot CLI (Claude Sonnet 4.5) improved from 66.70% to\n80.77%; and the Agent Framework (GPT-5) improved from 55.60% to 69.23%.<\/p>\n Model Selection<\/h3>\nClaude Sonnet 4.5 reached 80.77% accuracy vs. GPT-5 Mini at 69.23% on identical\ndatasets, prompts, and evaluation criteria in these final experiments. Our\ndefault and baseline model was GPT-5 Mini (we developed prompts there first);\nClaude generally performed better across many runs and produced more\ncomprehensive answers with the same prompt. This reflects model behavior on this\nsetup, not a universal ranking. Gemini 3.0 Pro Preview scored 74.07% on the same\nprompt and meta data. It showed strengths in whitespace\/temporal normalization\nand even surpassed ground truth quality in a few cases, but had gaps in\njoin-path coverage and JSON field selection. Models have distinct strengths and\nweaknesses, making it important to evaluate multiple options based on specific\nschema characteristics, latency\/cost constraints, and error tolerance\nrequirements.<\/p>\n Iterative Refinement and Feedback Loops<\/h3>\nBeyond initial prompt engineering and metadata setup, iterative feedback\nmechanisms unlock significant additional improvements. In our AI\/BI Genie\nevaluation, we found that storing corrected SQL patterns and learned joins from\nuser feedback improved accuracy by 19.27 percentage points (69.23% to 88.50%).\nThis progression demonstrates that NL-to-SQL systems benefit from production\nfeedback loops where successful query patterns are captured and reused,\nparticularly for repeated query types and domain-specific patterns. This\nsuggests that production systems should consider mechanisms to identify common\nqueries, validate results, and feed corrections back into the knowledge base for\ncontinuous improvement.<\/p>\n Genie’s feedback loop captures and reuses corrected queries, making it strong\nfor optimizing known or repeated queries, but applying those improvements is\nlargely manual; it lacks automation for end-to-end evaluation or broad data\nexploration, so it is less suited to hands-free exploration.<\/p>\n Business Logic Limitations<\/h3>\nThe majority of the remaining failures across all systems were business logic\nerrors: queries that executed successfully but returned incorrect results due to\nsemantic misunderstanding. This represents a fundamental challenge distinct from\ntechnical SQL generation.<\/p>\n Examples include:<\/p>\n \n- Incomplete cost aggregations (5 of 8 required fields, 46.00% underestimation).<\/li>\n
- Incorrect metric calculations (recomputing values rather than using\npre-calculated columns, which risked misalignment in calculation logic, units,\nor decimal precision).<\/li>\n
- Ambiguous interpretations (pattern matching “ultra-low temperature” vs.\nspecific “-70\u00b0C” value).<\/li>\n<\/ul>\n
We found that metadata and prompt optimization did not resolve business logic\nerrors. These require domain expertise, comprehensive test coverage, and\niterative refinement. We learned to encode formulas and constraints explicitly\nthrough prompt engineering or knowledge bases, rather than expecting models to\ninfer them.<\/p>\n Evaluation Strategy<\/h3>\nWhile our flexible matching approach handled numeric precision and row-level\nvariations, structural differences remained challenging. Queries returning\ndifferent numbers of columns, whitespace normalization, and semantic aliases\ncould make functionally equivalent results appear different. Adjusting\nevaluation for these raised accuracy significantly. In a few cases the\nprediction outperformed ground truth (e.g., better whitespace normalization and\ntemporal filtering). We learned that evaluation strategy design should occur\nearly in the process and be tuned to the specific domain (e.g., numeric\ntolerance for finance, column flexibility for messy schemas) to avoid marking\ncorrect solutions as failures.<\/p>\n False Negative Example<\/h4>\nWhen asked to calculate the percentage of returns with warranty claims, the\nground truth returned:<\/p>\n {\"wcr_percent\": \"53.30\"}<\/code><\/pre>\nWhile our agent returned:<\/p>\n {\"total_returns\": \"1500\", \"returns_with_warranty_claim\": \"799\", \"percent_with_warranty_claim\": \"53.30\"}<\/code><\/pre>\nThe percentage value was identical (53.30%), but the query was initially marked\nincorrect due to different column names and the inclusion of supporting detail\ncolumns. This highlights a fundamental evaluation challenge: we cannot simply\nmatch on column names because LLMs naturally generate different names than the\nground truth. We could have invested further. For example, using an LLM to infer\nmappings between expected and actual column names\u2014but we found that the accuracy\nscores we achieved were sufficiently good to demonstrate the effectiveness of\nagent-based approaches for NL-to-SQL tasks. Our evaluation strategy was\npragmatic: precise enough to validate the approach, without over-engineering\nbeyond what was needed for our conclusions.<\/p>\n Conclusion<\/h2>\nNL-to-SQL generation with AI agents can achieve high success rates with the\nright approach. Our experiments demonstrate that combining runtime data querying\nwith comprehensive schema information (both technical metadata and domain\ndocumentation) is critical\u2014without it, accuracy drops significantly (by 13-79\npercentage points depending on the system). Model choice matters substantially:\nClaude Sonnet 4.5 reached 80.77% vs. GPT-5 Mini at 69.23% on identical tasks.\nInvest time trialing models to find the right fit for your schema, latency\/cost\nconstraints, and error tolerance.<\/p>\n However, our work also revealed fundamental limitations, particularly the\nbusiness logic errors discussed earlier that require dedicated domain expertise\nto address.<\/p>\n Key takeaways for practitioners:<\/p>\n \n- Start with schema documentation and runtime validation<\/strong> – These provide\nthe highest ROI for accuracy improvements<\/li>\n
- Design evaluation early<\/strong> – Choose evaluation criteria that align with your\ndata characteristics and business goals. This ensures your experimentation\nactually measures what matters and reveals genuine improvements.<\/li>\n
- Expect business logic iteration<\/strong> – Budget for domain expert review and\ncontinuous refinement<\/li>\n
- Trial multiple models<\/strong> – Performance differences of 10-20 percentage\npoints are common<\/li>\n<\/ol>\n
NL-to-SQL is no longer a research problem, it’s an engineering challenge with\nknown solutions. The question is whether the remaining ~25.00% error rate fits\nyour use case, and whether you can iteratively address business logic gaps\nthrough domain knowledge and feedback loops.<\/p>\n","protected":false},"excerpt":{"rendered":" Evaluating AI agents for NL-to-SQL generation across Azure Databricks AI\/BI Genie, GitHub Copilot CLI, and Microsoft Agent Framework. We achieved ~75% accuracy with schema documentation and runtime validation, while discovering that business logic errors represent a fundamental limitation requiring domain expertise.<\/p>\n","protected":false},"author":181523,"featured_media":16636,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1,19],"tags":[3400],"class_list":["post-16635","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cse","category-machine-learning","tag-ise"],"acf":[],"blog_post_summary":" Evaluating AI agents for NL-to-SQL generation across Azure Databricks AI\/BI Genie, GitHub Copilot CLI, and Microsoft Agent Framework. We achieved ~75% accuracy with schema documentation and runtime validation, while discovering that business logic errors represent a fundamental limitation requiring domain expertise.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/16635","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/181523"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=16635"}],"version-history":[{"count":1,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/16635\/revisions"}],"predecessor-version":[{"id":16637,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/16635\/revisions\/16637"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media\/16636"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=16635"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=16635"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=16635"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
| | |
![\"agent-flow-diagram\"](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2026\/05\/agent-flow-diagram-scaled.webp\")
<\/p>\n