Azure OpenAI Chat Completion with data source provides powerful capabilities for integrating conversational AI into applications. However, using a data source and function calling in a single request is not supported yet. When both features are enabled, function calling is ignored, and only the data source is used. This presents a challenge when retrieving information, as a single request might not be sufficient to obtain an answer.

This article shows how to address this limitation with custom retry mechanism. If a query remains unanswered, the system sequentially retries with different sources until the requested information is found or a response is generated explaining its unavailability.

Solution overview

The solution involves:

1. Sending an initial request to Azure OpenAI without a data source or function calling enabled.
2. Evaluating whether the model’s response sufficiently answers the question.
3. If unanswered, retrying the query using an alternate approach:
   • If Azure Data Source was not used initially, enable it and retry.
   • If function calling was not used initially, enable it and retry.
4. Returning the final result once the query is successfully answered or when no further alternatives remain.

This approach ensures that the best available data is used dynamically without running into limitations.

Plugin Definition

The DataPlugin class provides user information and exposes a function for retrieving emails based on user names.

public sealed class DataPlugin
{
    private readonly Dictionary<string, string> _emails = new()
    {
        ["Emily"] = "emily@test.com",
        ["David"] = "david@test.com",
    };

    [KernelFunction]
    public List<string> GetEmails(List<string> users)
    {
        var emails = new List<string>();

        foreach (var user in users)
        {
            if (this._emails.TryGetValue(user, out var email))
            {
                emails.Add(email);
            }
        }

        return emails;
    }
}

ModelResult Definition

The ModelResult class is responsible for parsing and interpreting the AI model’s responses.

public sealed class ModelResult
{
    public string Message { get; set; }

    public bool IsAnswered { get; set; }

    public static ModelResult? Parse(string? result)
    {
        if (string.IsNullOrWhiteSpace(result))
        {
            return null;
        }

        // With response format as "json_object", sometimes the JSON response string is coming together with annotation.
        // The following line normalizes the response string in order to deserialize it later.
        var normalized = result
            .Replace("```json", string.Empty)
            .Replace("```", string.Empty);

        return JsonSerializer.Deserialize<ModelResult>(normalized);
    }
}

Retry Filter Definition

The FunctionInvocationRetryFilter class is responsible for retrying the request with different information sources. If the model does not answer the question, the filter first enables the Azure Data Source, and if that fails, function calling is enabled.

public sealed class FunctionInvocationRetryFilter : IFunctionInvocationFilter
{
    public async Task OnFunctionInvocationAsync(FunctionInvocationContext context, Func<FunctionInvocationContext, Task> next)
    {
        // Retry logic for Azure Data Source and function calling is enabled only for Azure OpenAI prompt execution settings.
        if (context.Arguments.ExecutionSettings is not null &&
            context.Arguments.ExecutionSettings.TryGetValue(PromptExecutionSettings.DefaultServiceId, out var executionSettings) &&
            executionSettings is AzureOpenAIPromptExecutionSettings azureOpenAIPromptExecutionSettings)
        {
            // Store the initial data source and function calling configuration to reset it after filter execution.
            var initialAzureChatDataSource = azureOpenAIPromptExecutionSettings.AzureChatDataSource;
            var initialFunctionChoiceBehavior = azureOpenAIPromptExecutionSettings.FunctionChoiceBehavior;

            // Track which source of information was used during the execution to try both sources sequentially.
            var dataSourceUsed = initialAzureChatDataSource is not null;
            var functionCallingUsed = initialFunctionChoiceBehavior is not null;

            // Perform a request.
            await next(context);

            // Get and parse the result.
            var result = context.Result.GetValue<string>();
            var modelResult = ModelResult.Parse(result);

            // If the model could not answer the question, then retry the request using an alternate technique:
            // - If the Azure Data Source was used then disable it and enable function calling.
            // - If function calling was used then disable it and enable the Azure Data Source.
            while (modelResult?.IsAnswered is false || (!dataSourceUsed && !functionCallingUsed))
            {
                // If Azure Data Source wasn't used - enable it.
                if (azureOpenAIPromptExecutionSettings.AzureChatDataSource is null)
                {
                    var dataSource = new AzureSearchChatDataSource
                    {
                        Endpoint = new Uri("data-source-endpoint"),
                        Authentication = DataSourceAuthentication.FromApiKey("data-source-api-key"),
                        IndexName = "data-source-index-name"
                    };

                    // Since Azure Data Source is enabled, the function calling should be disabled,
                    // because they are not supported together.
                    azureOpenAIPromptExecutionSettings.AzureChatDataSource = dataSource;
                    azureOpenAIPromptExecutionSettings.FunctionChoiceBehavior = null;

                    dataSourceUsed = true;
                }
                // Otherwise, if function calling wasn't used - enable it.
                else if (azureOpenAIPromptExecutionSettings.FunctionChoiceBehavior is null)
                {
                    // Since function calling is enabled, the Azure Data Source should be disabled,
                    // because they are not supported together.
                    azureOpenAIPromptExecutionSettings.AzureChatDataSource = null;
                    azureOpenAIPromptExecutionSettings.FunctionChoiceBehavior = FunctionChoiceBehavior.Auto();

                    functionCallingUsed = true;
                }

                // Perform a request.
                await next(context);

                // Get and parse the result.
                result = context.Result.GetValue<string>();
                modelResult = ModelResult.Parse(result);
            }

            // Reset prompt execution setting properties to the initial state.
            azureOpenAIPromptExecutionSettings.AzureChatDataSource = initialAzureChatDataSource;
            azureOpenAIPromptExecutionSettings.FunctionChoiceBehavior = initialFunctionChoiceBehavior;
        }
        // Otherwise, perform a default function invocation.
        else
        {
            await next(context);
        }
    }
}

Using Plugin and Retry Filter

The following method demonstrates how the plugin and retry filter work together to handle unanswered queries by switching between data source and function calling.

[Fact]
public async Task TestAsync()
{
    var builder = Kernel.CreateBuilder()
        .AddAzureOpenAIChatCompletion(
            "deployment-name",
            "endpoint",
            "api-key");

    // Add retry filter.
    builder.Services.AddSingleton<IFunctionInvocationFilter, FunctionInvocationRetryFilter>();

    var kernel = builder.Build();

    // Import plugin.
    kernel.ImportPluginFromType<DataPlugin>();

    // Define response schema.
    // The model evaluates its own answer and provides a boolean flag,
    // which allows to understand whether the user's question was actually answered or not.
    // Based on that, it's possible to make a decision whether the source of information should be changed or the response
    // should be provided back to the user.
    var responseSchema =
        """
        {
            "type": "object",
            "properties": {
                "Message": { "type": "string" },
                "IsAnswered": { "type": "boolean" },
            }
        }
        """;

    // Define execution settings with response format and initial instructions.
    var promptExecutionSettings = new AzureOpenAIPromptExecutionSettings
    {
        ResponseFormat = "json_object",
        ChatSystemPrompt =
            "Provide concrete answers to user questions. " +
            "If you don't have the information - do not generate it, but respond accordingly. " +
            $"Use following JSON schema for all the responses: {responseSchema}. "
    };

    // First question without previous context based on uploaded content.
    var ask = "How did Emily and David meet?";

    // The answer to the first question is expected to be fetched from Azure Data Source (in this example Azure AI Search).
    // Azure Data Source is not enabled in initial execution settings, but is configured in retry filter.
    var response = await kernel.InvokePromptAsync(ask, new(promptExecutionSettings));
    var modelResult = ModelResult.Parse(response.ToString());

    // Output
    // Ask: How did Emily and David meet?
    // Response: Emily and David, both passionate scientists, met during a research expedition to Antarctica [doc1].
    Console.WriteLine($"Ask: {ask}");
    Console.WriteLine($"Response: {modelResult?.Message}");

    ask = "Can I have Emily's and David's emails?";

    // The answer to the second question is expected to be fetched from DataPlugin-GetEmails function using function calling.
    // Function calling is not enabled in initial execution settings, but is configured in retry filter.
    response = await kernel.InvokePromptAsync(ask, new(promptExecutionSettings));
    modelResult = ModelResult.Parse(response.ToString());

    // Output
    // Ask: Can I have their emails?
    // Response: Emily's email is emily@test.com and David's email is david@test.com.
    Console.WriteLine($"Ask: {ask}");
    Console.WriteLine($"Response: {modelResult?.Message}");
}

This structured approach ensures that if a query remains unanswered, the system intelligently retries with different sources, providing a more reliable conversational AI experience.

Summary

Provided example demonstrates how to enable both Azure OpenAI Chat Completion data source and function calling to answer user questions using a retry mechanism.

We’re always interested in hearing from you. If you have feedback, questions or want to discuss further, feel free to reach out to us and the community on the discussion boards on GitHub! We would also love your support, if you’ve enjoyed using Semantic Kernel, give us a star on GitHub.