OpenAI recently introduced a powerful new feature for developers: structured output using JSON Schema via the response_format parameter. This makes it possible to request responses from a GPT-4o model that strictly match a given schema—no free-text, no guesswork. If you’re working with Azure SQL, this is a game-changer. Combined with the sp_invoke_external_rest_endpoint stored procedure and SQL Server’s built-in JSON functions, you can now:

• Call an LLM directly from T-SQL
• Receive output that strictly matches a schema you define
• Parse that output into a SQL table

What Is Structured Output?

By default, LLMs like GPT-4 return free-form text—which can be difficult to parse reliably. With structured output, you provide a JSON Schema along with your request, and the model is implicitly instructed to generate a response that strictly adheres to that format. For example, if you want to have an output that would look like the following result:

You have to provide the following JSON schema:

declare @js nvarchar(max) = N'{
    "type": "json_schema",
    "json_schema": {
        "name": "products",
        "strict": true,
        "schema": {
            "type": "object",
            "properties": {
                "products": {
                    "type": "array",
                    "items": {
                        "type": "object",
                        "properties": {
                            "result_position": {
                                "type": "number"
                            },
                            "id": {
                                "type": "number"
                            },
                            "description": {
                                "type": "string",
                                "description": "a brief and summarized description of the product, no more than ten words"
                            },                            
                            "thoughts": {
                                "type": "string",
                                "description": "explanation of why the product has been chosen"
                            }
                        },
                        "required": [
                            "result_position",
                            "id",                            
                            "description",                            
                            "thoughts"                            
                        ],
                        "additionalProperties": false
                    }
                }
            },
            "required": ["products"],
            "additionalProperties": false
        }        
    }        
}'

How to Use It in Azure SQL

Nothing real new under the sun here. Just use the usual sp_invoke_external_rest_endpoint to call your favorite LLM (as long as it supports structured output) and provide the JSON Schema in the response_format property in the sent JSON payload:

set @prompt = json_modify(@prompt, '$.response_format', json_query(@js))

then make the call:

declare @retval int, @response nvarchar(max);
exec @retval = sp_invoke_external_rest_endpoint
    @url = '<OPENAI_URL>/openai/deployments/gpt-4o/chat/completions?api-version=2024-08-01-preview',
    @headers = '{"Content-Type":"application/json"}',
    @method = 'POST',
    @credential = [<OPENAI_URL>],
    @timeout = 120,
    @payload = @prompt,
    @response = @response output
    with result sets none;

and then all you have to do is take the resulting output and parse it into a table:

select 
    sr.* 
from 
    #r
cross apply
    openjson(response, '$.result.choices[0].message') with (
        content nvarchar(max) '$.content'
    ) m
cross apply
    openjson(m.content, '$.products') with (
        result_position int,
        id int,        
        [description] nvarchar(max),
        thoughts nvarchar(max)
    ) as sr

Pretty easy!

Why This Is a Big Deal

This structured approach solves a core problem: predictability. In fact, one of the biggest challenges when working with large language models is the unpredictability of their responses. While free-form text might be easy for humans to understand, it’s often inconsistent and unreliable when you need to process it programmatically. If you’re trying to build systems that depend on LLM-generated content—especially within structured environments like databases—this lack of consistency becomes a serious bottleneck.

Why? Because handling that unpredictability in code is hard. You end up writing custom parsing logic, regular expressions, or brittle string manipulation scripts that are difficult to maintain and easy to break. Every time the model returns a slightly different phrasing or changes the order of elements, your code risks falling apart. It creates technical debt fast and slows down development, making it harder to build reliable, production-grade systems on top of AI.

That’s where structured output changes the game. By supplying a JSON Schema along with your prompt, you’re telling the model exactly what format the output should take. And with strict: true, the model will stick to that structure rigidly. No more guessing what the model meant. No more fragile parsing logic. Just clean, well-formed JSON that you can immediately trust and use.

For developers working with Azure SQL or SQL Server, this predictability is powerful. You can take the JSON result from the model and easily map it into a table using SQL Server’s native JSON functions. This means you can bring AI insights directly into your SQL workflows—filtering, aggregating, joining with other datasets—just like any other data. It effectively closes the loop between natural language and structured data processing.

What if you’re not using OpenAI models?

What if you want to host a LLM model on your own? Can you use structured output in that case? Then answer is most likely “yes”, provided that you are serving an LLM model capable of structured output. Then, inference servers like Ollama or vLLM all added support to structured output to their API:

• Ollama Structured Outputs
• vLLM Structured Outputs

The way you’ll notify the model that you want to have the results returned using a specified JSON schema is different based on the inference server you’re using, but the principle is similar to the one described for OpenAI.

Final Thoughts

The combination of structured output and Azure SQL’s REST and JSON support unlocks an entirely new kind of intelligent data workflow. With this powerful integration, you can build natural-language interfaces on top of your data, allowing for more intuitive and user-friendly interactions. Additionally, you can enrich your reports with AI-generated insights, providing deeper and more meaningful analysis. Furthermore, you can generate summaries, tags, classifications, and more—all directly from SQL, streamlining your data processes and enhancing overall efficiency.

Check out the working sample: https://github.com/Azure-Samples/azure-sql-db-chatbot