![\"OpenAI](\"https:\/\/devblogs.microsoft.com\/semantic-kernel\/wp-content\/uploads\/sites\/78\/2025\/04\/ghibli-audio-preview.jpg\")
<\/a><\/center>OpenAI’s gpt-4o-audio-preview<\/strong>\u00a0is a powerful multimodal model that enables audio input and output capabilities, allowing developers to create more natural and accessible AI interactions. This model supports both speech-to-text and text-to-speech functionalities in a single API call through the Chat Completions API, making it suitable for building voice-enabled applications where turn-based interactions are appropriate.<\/p>\nIn this post, we’ll explore how to use the audio-preview model with Semantic Kernel in both C# and Python to create voice-enabled AI applications.<\/p>\n
Best Use Cases<\/h3>\n
Best for turn-based interactions where complete audio messages are processed as discrete units. Suitable for applications like voice-based Q&A systems, audio transcription with AI responses, or asynchronous voice messaging where real-time interaction isn’t critical.<\/p>\n
Key Features of OpenAI’s Audio-Preview Model with Chat Completions API<\/h2>\n\n- \n
Multimodal Input\/Output<\/strong>: Process both text and audio inputs, and generate both text and audio outputs in a single API call.<\/p>\n<\/li>\n- \n
Turn-Based Voice Interactions<\/strong>: Suitable for non-real-time, turn-based conversational applications where each interaction is a complete request-response cycle.<\/p>\n<\/li>\n- \n
Voice Synthesis Options<\/strong>: Generate speech with support for multiple voices (like Alloy, Echo, Fable, Onyx, Nova, and Shimmer).<\/p>\n<\/li>\n- \n
Audio Understanding<\/strong>: Transcribe and comprehend spoken language from audio files.<\/p>\n<\/li>\n- \n
Multilingual Support<\/strong>: Process and generate audio in multiple languages, making it accessible to global users.<\/p>\n<\/li>\n- \n
Integration with Function Calling<\/strong>: Combine audio capabilities with function calling to create voice-controlled applications that can perform actions.<\/p>\n<\/li>\n- \n
Simplified Development<\/strong>: Single API for both audio input processing and audio output generation, reducing the complexity of building voice-enabled applications.<\/p>\n<\/li>\n- \n
Batch Processing<\/strong>: Well-suited for applications where complete audio messages are processed as discrete units rather than continuous streams.<\/p>\n<\/li>\n<\/ul>\nNote<\/strong>: For truly low-latency, real-time voice interactions, OpenAI’s Realtime API is the more appropriate choice. The Chat Completions API with audio capabilities is better suited for non-real-time applications where some latency is acceptable.<\/p>\nUsing Audio-Preview in Semantic Kernel<\/h2>\n
Semantic Kernel provides a straightforward way to integrate with OpenAI’s audio-preview model. Let’s see how to implement basic audio input and output functionality in both C# and Python.<\/p>\n
In .NET (C#)<\/h3>\n
For a C# project using Semantic Kernel, you can add the audio-preview model as an OpenAI chat completion service. Make sure you have your OpenAI API key (or Azure OpenAI endpoint and key if using Azure):<\/p>\n
using<\/span> Microsoft.SemanticKernel;\r\nusing<\/span> Microsoft.SemanticKernel.ChatCompletion;\r\nusing<\/span> Microsoft.SemanticKernel.Connectors.OpenAI;\r\n\r\n\/\/ Initialize the OpenAI chat completion service with the audio-preview model<\/span>\r\nvar<\/span> kernel = Kernel.CreateBuilder()\r\n .AddOpenAIChatCompletion(\r\n modelId: \"gpt-4o-audio-preview\"<\/span>,\r\n apiKey: \"YOUR_OPENAI_API_KEY\"<\/span>\r\n )\r\n .Build();\r\n\r\nvar<\/span> chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();\r\n\r\n\/\/ Configure settings for audio output<\/span>\r\nvar<\/span> settings = new<\/span> OpenAIPromptExecutionSettings\r\n{\r\n Audio = new<\/span> ChatAudioOptions(\r\n ChatOutputAudioVoice.Shimmer, \/\/ Choose from available voices<\/span>\r\n ChatOutputAudioFormat.Mp3 \/\/ Choose output format<\/span>\r\n ),\r\n Modalities = ChatResponseModalities.Text | ChatResponseModalities.Audio \/\/ Request both text and audio<\/span>\r\n};\r\n\r\n\/\/ Create a chat history and add an audio message<\/span>\r\nvar<\/span> chatHistory = new<\/span> ChatHistory(\"You are a helpful assistant.\"<\/span>);\r\n\r\n\/\/ Add audio input (from a file or recorded audio)<\/span>\r\nbyte<\/span>[] audioBytes = File.ReadAllBytes(\"user_question.wav\"<\/span>);\r\nchatHistory.AddUserMessage([new<\/span> AudioContent(audioBytes, \"audio\/wav\"<\/span>)]);\r\n\r\n\/\/ Get the model's response with both text and audio<\/span>\r\nvar<\/span> result = await<\/span> chatCompletionService.GetChatMessageContentAsync(chatHistory, settings);\r\n\r\n\/\/ Access the text response<\/span>\r\nConsole.WriteLine($\"Assistant > {result}<\/span>\"<\/span>);\r\n\r\n\/\/ Access the audio response (if available)<\/span>\r\nif<\/span> (result.Items.OfType<AudioContent>().Any())\r\n{\r\n var<\/span> audioContent = result.Items.OfType<AudioContent>().First();\r\n \/\/ Save or play the audio response<\/span>\r\n File.WriteAllBytes(\"assistant_response.mp3\"<\/span>, audioContent.Data.ToArray());\r\n}\r\n<\/code><\/pre>\nWe have also created a C# sample within Semantic Kernel repository using audio-preview model here:<\/p>\n
\n- Chat Completion with Audio<\/a><\/li>\n<\/ul>\n
Conclusion<\/h2>\n
OpenAI’s audio-preview model represents a significant advancement in creating more natural and accessible AI interactions. With Semantic Kernel’s straightforward integration, developers can build voice-enabled applications that provide an enhanced user experience.<\/p>\n
References<\/h2>\n\n- OpenAI Platform Documentation – Audio and Speech<\/a><\/li>\n
- Azure OpenAI Service Models<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"
OpenAI’s gpt-4o-audio-preview\u00a0is a powerful multimodal model that enables audio input and output capabilities, allowing developers to create more natural and accessible AI interactions. This model supports both speech-to-text and text-to-speech functionalities in a single API call through the Chat Completions API, making it suitable for building voice-enabled applications where turn-based interactions are appropriate. In this […]<\/p>\n","protected":false},"author":63983,"featured_media":4602,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[78,27,2,1],"tags":[79,48,49,69,31,63,9],"class_list":["post-4601","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-net","category-agents","category-samples","category-semantic-kernel","tag-net","tag-ai","tag-ai-agents","tag-api","tag-c","tag-microsoft-semantic-kernel","tag-semantic-kernel"],"acf":[],"blog_post_summary":"
OpenAI’s gpt-4o-audio-preview\u00a0is a powerful multimodal model that enables audio input and output capabilities, allowing developers to create more natural and accessible AI interactions. This model supports both speech-to-text and text-to-speech functionalities in a single API call through the Chat Completions API, making it suitable for building voice-enabled applications where turn-based interactions are appropriate. In this […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/4601","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/users\/63983"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/comments?post=4601"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/4601\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media\/4602"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media?parent=4601"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/categories?post=4601"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/tags?post=4601"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Multimodal Input\/Output<\/strong>: Process both text and audio inputs, and generate both text and audio outputs in a single API call.<\/p>\n<\/li>\n Turn-Based Voice Interactions<\/strong>: Suitable for non-real-time, turn-based conversational applications where each interaction is a complete request-response cycle.<\/p>\n<\/li>\n Voice Synthesis Options<\/strong>: Generate speech with support for multiple voices (like Alloy, Echo, Fable, Onyx, Nova, and Shimmer).<\/p>\n<\/li>\n Audio Understanding<\/strong>: Transcribe and comprehend spoken language from audio files.<\/p>\n<\/li>\n Multilingual Support<\/strong>: Process and generate audio in multiple languages, making it accessible to global users.<\/p>\n<\/li>\n Integration with Function Calling<\/strong>: Combine audio capabilities with function calling to create voice-controlled applications that can perform actions.<\/p>\n<\/li>\n Simplified Development<\/strong>: Single API for both audio input processing and audio output generation, reducing the complexity of building voice-enabled applications.<\/p>\n<\/li>\n Batch Processing<\/strong>: Well-suited for applications where complete audio messages are processed as discrete units rather than continuous streams.<\/p>\n<\/li>\n<\/ul>\n Note<\/strong>: For truly low-latency, real-time voice interactions, OpenAI’s Realtime API is the more appropriate choice. The Chat Completions API with audio capabilities is better suited for non-real-time applications where some latency is acceptable.<\/p>\n Semantic Kernel provides a straightforward way to integrate with OpenAI’s audio-preview model. Let’s see how to implement basic audio input and output functionality in both C# and Python.<\/p>\n For a C# project using Semantic Kernel, you can add the audio-preview model as an OpenAI chat completion service. Make sure you have your OpenAI API key (or Azure OpenAI endpoint and key if using Azure):<\/p>\n We have also created a C# sample within Semantic Kernel repository using audio-preview model here:<\/p>\n OpenAI’s audio-preview model represents a significant advancement in creating more natural and accessible AI interactions. With Semantic Kernel’s straightforward integration, developers can build voice-enabled applications that provide an enhanced user experience.<\/p>\n OpenAI’s gpt-4o-audio-preview\u00a0is a powerful multimodal model that enables audio input and output capabilities, allowing developers to create more natural and accessible AI interactions. This model supports both speech-to-text and text-to-speech functionalities in a single API call through the Chat Completions API, making it suitable for building voice-enabled applications where turn-based interactions are appropriate. In this […]<\/p>\n","protected":false},"author":63983,"featured_media":4602,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[78,27,2,1],"tags":[79,48,49,69,31,63,9],"class_list":["post-4601","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-net","category-agents","category-samples","category-semantic-kernel","tag-net","tag-ai","tag-ai-agents","tag-api","tag-c","tag-microsoft-semantic-kernel","tag-semantic-kernel"],"acf":[],"blog_post_summary":" OpenAI’s gpt-4o-audio-preview\u00a0is a powerful multimodal model that enables audio input and output capabilities, allowing developers to create more natural and accessible AI interactions. This model supports both speech-to-text and text-to-speech functionalities in a single API call through the Chat Completions API, making it suitable for building voice-enabled applications where turn-based interactions are appropriate. In this […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/4601","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/users\/63983"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/comments?post=4601"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/4601\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media\/4602"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media?parent=4601"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/categories?post=4601"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/tags?post=4601"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Using Audio-Preview in Semantic Kernel<\/h2>\n
In .NET (C#)<\/h3>\n
using<\/span> Microsoft.SemanticKernel;\r\nusing<\/span> Microsoft.SemanticKernel.ChatCompletion;\r\nusing<\/span> Microsoft.SemanticKernel.Connectors.OpenAI;\r\n\r\n\/\/ Initialize the OpenAI chat completion service with the audio-preview model<\/span>\r\nvar<\/span> kernel = Kernel.CreateBuilder()\r\n .AddOpenAIChatCompletion(\r\n modelId: \"gpt-4o-audio-preview\"<\/span>,\r\n apiKey: \"YOUR_OPENAI_API_KEY\"<\/span>\r\n )\r\n .Build();\r\n\r\nvar<\/span> chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();\r\n\r\n\/\/ Configure settings for audio output<\/span>\r\nvar<\/span> settings = new<\/span> OpenAIPromptExecutionSettings\r\n{\r\n Audio = new<\/span> ChatAudioOptions(\r\n ChatOutputAudioVoice.Shimmer, \/\/ Choose from available voices<\/span>\r\n ChatOutputAudioFormat.Mp3 \/\/ Choose output format<\/span>\r\n ),\r\n Modalities = ChatResponseModalities.Text | ChatResponseModalities.Audio \/\/ Request both text and audio<\/span>\r\n};\r\n\r\n\/\/ Create a chat history and add an audio message<\/span>\r\nvar<\/span> chatHistory = new<\/span> ChatHistory(\"You are a helpful assistant.\"<\/span>);\r\n\r\n\/\/ Add audio input (from a file or recorded audio)<\/span>\r\nbyte<\/span>[] audioBytes = File.ReadAllBytes(\"user_question.wav\"<\/span>);\r\nchatHistory.AddUserMessage([new<\/span> AudioContent(audioBytes, \"audio\/wav\"<\/span>)]);\r\n\r\n\/\/ Get the model's response with both text and audio<\/span>\r\nvar<\/span> result = await<\/span> chatCompletionService.GetChatMessageContentAsync(chatHistory, settings);\r\n\r\n\/\/ Access the text response<\/span>\r\nConsole.WriteLine($\"Assistant > {result}<\/span>\"<\/span>);\r\n\r\n\/\/ Access the audio response (if available)<\/span>\r\nif<\/span> (result.Items.OfType<AudioContent>().Any())\r\n{\r\n var<\/span> audioContent = result.Items.OfType<AudioContent>().First();\r\n \/\/ Save or play the audio response<\/span>\r\n File.WriteAllBytes(\"assistant_response.mp3\"<\/span>, audioContent.Data.ToArray());\r\n}\r\n<\/code><\/pre>\n\n
Conclusion<\/h2>\n
References<\/h2>\n
\n