Today, more and more customers are embracing AI with one clear goal in mind: they want to chat with their data. And by “data,” they mean all of it:\u00a0 structured tables, unstructured documents, images, audio, and more.<\/p>\n
This demand has given rise to hybrid approaches that blend semantic understanding with precise data retrieval. But here\u2019s the challenge: while embeddings and vector search play a critical role in semantic search, they\u2019re not enough on their own \u2014 especially when the question is highly specific and demands an exact answer.<\/p>\n
As I discussed in Improve the \u201cR\u201d in RAG and Embrace Agentic RAG in Azure SQL<\/a> article, a smarter, multi-step approach is needed.<\/p>\n There\u2019s a code sample waiting for you at the end\u2014but don\u2019t scroll just yet! This article lays the foundation you\u2019ll need to really understand the strategy behind it. That way, you\u2019ll be able to apply the same pattern to your own data, not just the example. So read through it all\u2026 and no, asking an AI to summarize it won\u2019t cut it this time \ud83d\ude09<\/p>\n When users ask a question, it typically falls into one of two categories:<\/p>\n This post focuses on the second case, <\/strong>as it’s arguably the most complex.<\/p>\n Getting a precise, correct answer from a database using natural language is where LLMs struggle the most. They’re inherently designed to provide semantically relevant responses, not necessarily factually or structurally accurate ones. That\u2019s why this problem requires a carefully crafted multi-step approach, where the LLM is guided to do what it does best while minimizing the risk of error.<\/p>\n So how do you apply this to a real-world enterprise database with hundreds of tables?<\/p>\n The classic strategy of “divide and conquer” is your best friend here. You can\u2019t simply dump the full schema of 100+ tables into a large language model (LLM) and expect magic. LLMs have limitations \u2014 especially when it comes to context window size and relevance filtering.<\/p>\n The more irrelevant data you include, the more noise you’ll actually generate and so the less effective the model becomes. If you\u2019re asking a question about product pricing, providing the LLM with address type definitions only adds confusion. So, the problem needs to be solved with two separate steps: first, help the LLM to figure out what table are really needed, second help the LLM model to have detailed understanding of the schema of those tables, to that the best SQL query can be generate.<\/p>\n Start by giving the LLM a high-level map of your schema. Not every column, not every data type \u2014 just the purpose of each table.<\/p>\n Here\u2019s an example from the AdventureWorks2022 database, that can be used to define the prompt for the first agent:<\/p>\n The prompt would look like the following:<\/p>\n With this context, you ask the LLM to identify the most relevant tables needed to answer a specific question. Think of the LLM in this phase as a knowledgeable data architect \u2014 someone who knows where the data lives, but not exactly how it\u2019s structured inside each table.<\/strong><\/p>\n Once you have the list of candidate tables, then \u2014 and only then \u2014 you provide full schema details: column names, types, relationships, and even quirks like embedded JSON blobs in \u201cdetails\u201d columns.<\/p>\n This is your opportunity to give the LLM the context it actually needs, and nothing more. If a column holds JSON, you should explain its structure and provide examples of how to extract data from it.<\/p>\n Here\u2019s a sample of how you might present that information to the LLM for the In this second agent, the prompt is the following:<\/p>\n This type of structured metadata, minimal yet meaningful, gives the LLM enough signal to reason about what this table is for, without overwhelming it with noise. You can then repeat the process for other selected tables, tailoring the level of detail based on what the question requires.<\/p>\n By feeding the model only relevant, targeted information like this, you dramatically reduce confusion and increase the accuracy of the generated SQL query. You have now created a SQL expert that knows exactly how to query the involved tables.<\/p>\n This is where the Agentic approach shines. Think of this system as having two expert agents, each with a specific role:<\/p>\n Here\u2019s how a real interaction might play out:<\/p>\n Human:<\/strong> Hi, I want to know who is the most profitable customer.<\/p>\n Agent1:<\/strong> I need to query the database using [Sales].[Customer], [Sales].[SalesOrderHeader], [Sales].[SalesOrderDetail]. <\/em>Asking Agent2<\/em> to: “Identify the customer who has contributed the highest total profit. Sum up the profit for each customer using order amounts and subtracting related costs if available. Return the most profitable customer details including customer ID, name if available, and total profit.”<\/p>\n Agent2<\/strong>: Running the following query:<\/p>\n Answer<\/strong>: The most profitable customer is the one associated with Customer ID 29818 and Account Number AW00029818. This customer has contributed a total profit of approximately $877,107.19 to the business. If you need further details about this customer, let me know!<\/p>\n Human<\/strong>: Can you do a breakdown per year and then also return to me the name and some details about the customer?<\/p>\n Agent1<\/strong>: Use the tables: [Sales].[Customer], [Sales].[SalesOrderHeader], [Sales].[SalesOrderDetail]<\/em>. Asking Agent2<\/em> to: “Break down the total profit data of Customer ID 29818 by year and provide yearly profit details for this customer.”<\/p>\n Agent2:<\/strong> <Generates and runs the SQL query to return yearly profit breakdown><\/em><\/p>\n Agent1<\/strong>: Agent2<\/em>, now use [Sales].[Customer], [Person].[Person]<\/em> to: “Provide the name and additional details of the customer with Customer ID 29818, including contact or relevant information.”<\/p>\n Agent2:<\/strong> <Generates and runs the SQL to return customer name, email, etc.><\/em><\/p>\n Answer:<\/strong> <final answer><\/em><\/p>\n This flow mirrors real-world collaboration: schema design and data retrieval are distinct tasks, and each expert focuses on what they do best. The LLMs emulate that by acting as coordinated, specialized agents.<\/p>\n The approach we\u2019ve outlined, combining schema-aware reasoning with agentic collaboration, is a powerful way to enable natural language access to structured data. But in most real-world scenarios, structured data is only part of the story.<\/p>\n To truly \u201cchat with your data,\u201d you need to handle both structured and unstructured sources \u2014 and that\u2019s where RAG pattern<\/strong> comes in.<\/p>\n In practice, this means your system needs to make an early decision: Is this question best answered with a SQL query, or does it require semantic search over documents, images, or other unstructured content?<\/em><\/p>\n This decision is the first and most critical task for the LLM. As discussed in Improve the \u201cR\u201d in RAG and Embrace Agentic RAG in Azure SQL<\/a>, the best systems combine both approaches:<\/p>\n This hybrid model ensures that each question is routed to the right tool for the job, maximizing both accuracy and relevance.<\/p>\n Conversational access to enterprise data isn\u2019t just about writing better SQL or retrieving better documents. It\u2019s about orchestrating the right tools in the right order. That\u2019s where agentic patterns shine. By combining classic RAG with structured query generation, and by guiding LLMs through a multi-step reasoning process, you can build systems that are not only intelligent \u2014 but also trustworthy, scalable, and enterprise-ready.<\/p>\n Let\u2019s make data talk \u2014 no matter what form it takes.<\/p>\n The source code, as usual, is available on GitHub, here: https:\/\/github.com\/Azure-Samples\/azure-sql-nl2sql.<\/a><\/p>\n The sample is on purpose super simple, to make it easier for everyone to understand what’s going on. An improvement would be to use structured output to return the generated SQL query to make it easier to execute it. Execution of generated SQL of course, is done without any security check which is obviously not a best practice: in a real world deployment, the generate SQL *must* be sanitized and checked before executing it. I would also strongly recommend making sure you have all the proper security configuration done in the database, especially including Row-Level Security<\/a>, to avoid any possible prompt injection attack that could cause data leaks or worse. Having said that, have fun with the code, it is going to make your day, I promise!<\/p>\n","protected":false},"excerpt":{"rendered":" Today, more and more customers are embracing AI with one clear goal in mind: they want to chat with their data. And by “data,” they mean all of it:\u00a0 structured tables, unstructured documents, images, audio, and more. This demand has given rise to hybrid approaches that blend semantic understanding with precise data retrieval. But here\u2019s […]<\/p>\n","protected":false},"author":24720,"featured_media":4811,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[601,1],"tags":[673,590,674,613,627,486],"class_list":["post-4807","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","category-azure-sql","tag-agents","tag-ai","tag-natural-language","tag-nl2sql","tag-rag","tag-sql"],"acf":[],"blog_post_summary":" Today, more and more customers are embracing AI with one clear goal in mind: they want to chat with their data. And by “data,” they mean all of it:\u00a0 structured tables, unstructured documents, images, audio, and more. This demand has given rise to hybrid approaches that blend semantic understanding with precise data retrieval. But here\u2019s […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/posts\/4807","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/users\/24720"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/comments?post=4807"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/posts\/4807\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/media\/4811"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/media?parent=4807"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/categories?post=4807"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/tags?post=4807"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Two Kinds of Questions, Two Kinds of Answers<\/h2>\n
\n
\u201cDivide and Conquer\u201d Still Works<\/h2>\n
Agent 1: Create a high-level, domain expert<\/h3>\n
[Production].[ProductListPriceHistory]: Changes in the list price of a product over time.\r\n[Person].[Address]: Street address information for customers, employees, and vendors.\r\n[Sales].[SpecialOfferProduct]: Cross-reference table mapping products to special offer discounts.\r\n[Production].[ProductModel]: Product model classification.\r\n[Person].[AddressType]: Types of addresses stored in the Address table.\r\n[Person].[StateProvince]: State and province lookup table.<\/code><\/pre>\nYou are an AI assistant that helps users to query the database. The tables in the database, with the related description, are:\r\n\r\n{await database.GetTablesAsync()} \r\n\r\nUse a professional tone when aswering and provide a summary of data instead of lists. \r\nIf users ask about topics you don't know, answer that you don't know. Today's date is {DateTime.Now:yyyy-MM-dd}. \r\nYou must answer providing a list of tables that must be used to answer the question and an explanation of that you'll be doing to answer the question.\r\nYou must use the provided tool to query the database.\r\nIf the request is complex, break it down into smaller steps and call the plugin as many time as needed. Ideally don't use most than 5 tables in the same query. \r\n<\/code><\/pre>\nAgent 2: Create a SQL expert<\/h3>\n
[Sales].[SalesTerritory]<\/code> table:<\/p>\n[TerritoryID] int -- Primary key for SalesTerritory records.\r\n[Name] Name -- Sales territory description\r\n[CountryRegionCode] nvarchar -- ISO standard country or region code. Foreign key to CountryRegion.CountryRegionCode.\r\n[Group] nvarchar -- Geographic area to which the sales territory belongs.<\/code><\/pre>\nYou create T-SQL queries based on the given user request and the provided schema. Just return T-SQL query to be executed. \r\nDo not return other text or explanation. Don't use markdown or any wrappers.\r\nThe schema is provided in the format: \r\n \r\nTable1Name: \r\nColumn1Name (Column1Type) -- Column1Description\r\nColumn2Name (Column2Type) -- Column2Description\r\n...\r\nColumnNName (ColumnNType) -- ColumnNDescription\r\n\r\nTable2Name: \r\nColumn1Name (Column1Type) -- Column1Description\r\nColumn2Name (Column2Type) -- Column2Description\r\n...\r\nColumnNName (ColumnNType) -- ColumnNDescription\r\n\r\nThe schema for the avaiable tables is the following:\r\n \r\n{table_schemas.ToString()}\r\n \r\nGenerate the T-SQL query based on the provided schema and the user request. The user request is in the next message.<\/code><\/pre>\nMaking the Agents Working Together<\/h3>\n
\n
SELECT TOP 1\r\n C.CustomerID,\r\n C.AccountNumber,\r\n SUM(SOD.LineTotal) AS TotalProfit\r\nFROM\r\n [Sales].[Customer] AS C\r\nJOIN\r\n [Sales].[SalesOrderHeader] AS SOH ON C.CustomerID = SOH.CustomerID\r\nJOIN\r\n [Sales].[SalesOrderDetail] AS SOD ON SOH.SalesOrderID = SOD.SalesOrderID\r\nGROUP BY\r\n C.CustomerID, C.AccountNumber\r\nORDER BY\r\n TotalProfit DESC;<\/code><\/pre>\nWhat\u2019s Next: From Queries to Conversations<\/h2>\n
\n
Final Word<\/h2>\n
Try it yourself<\/h2>\n