{"id":3645,"date":"2024-11-15T08:49:46","date_gmt":"2024-11-15T16:49:46","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/semantic-kernel\/?p=3645"},"modified":"2024-11-15T09:18:32","modified_gmt":"2024-11-15T17:18:32","slug":"working-with-audio-in-semantic-kernel-python","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/agent-framework\/working-with-audio-in-semantic-kernel-python\/","title":{"rendered":"Working with Audio in Semantic Kernel Python"},"content":{"rendered":"

We are pleased to announce the arrival of audio support in Semantic Kernel Python. This new audio functionality will enable you to create more interactive and accessible user experiences. In this blog post, I will detail the new interface, the existing connectors, and provide samples. Please continue reading for more information.<\/span><\/p>\n

Audio-to-Text<\/span><\/h3>\n

The first feature we are introducing is the ability to transcribe audio into text. The foundational class for this feature is\u00a0AudioToTextClientBase<\/code><\/strong>, which includes two public methods:\u00a0get_text_contents<\/code>\u00a0<\/strong>and\u00a0get_text_content<\/code><\/strong>. The former returns a list of possible transcriptions based on the number requested, while the latter returns a single transcription. The transcriptions are returned as TextContent<\/code> <\/strong>objects. Notably, get_text_content<\/code>\u00a0<\/strong>internally calls\u00a0get_text_contents<\/code>\u00a0<\/strong>and simply returns the first transcription from the list.<\/span><\/p>\n

As of the publication of this blog post, the available services include\u00a0OpenAIAudioToText<\/code>\u00a0<\/strong>and\u00a0AzureAudioToText<\/code><\/strong>, allowing you to utilize either your OpenAI endpoints or Azure deployments.<\/span><\/p>\n

AudioContent<\/span><\/h4>\n

With the introduction of audio support, we are also introducing a new content type known as\u00a0AudioContent<\/code><\/strong>. Instances of the\u00a0AudioContent<\/code>\u00a0<\/strong>class should encapsulate either the binary data or the URI pointing to the location of the audio data. Additionally, this class offers a convenient method that allows you to create an\u00a0AudioContent<\/code> <\/strong>object directly from a file:<\/span><\/p>\n

AudioContent.from_audio_file(path=PATH_TO_AUDIO_FILE)<\/code><\/pre>\n
Audio-to-Text Example Using Azure OpenAI<\/span><\/h4>\n
Please ensure that Semantic Kernel is updated to the latest version. To process audio input, the following components are required:<\/span><\/p>\n
    \n
  1. A speech-to-text model, such as whisper-1<\/span><\/li>\n
  2. An audio input device<\/span><\/li>\n<\/ol>\n
    To begin, create the service with the following code:<\/span><\/p>\n
    \n
    \n
    from semantic_kernel.connectors.ai.open_ai.services.azure_audio_to_text import AzureAudioToText\r\n\r\n\r\naudio_to_text_service = AzureAudioToText(api_key=\"...\", deployment_name=\"...\", endpoint=\"...\")<\/code><\/pre>\n<\/div>\n<\/div>\n
    Next, you will need to create the audio content that will be transcribed into text:<\/span><\/p>\n
    from semantic_kernel.contents.audio_content import AudioContent\r\n\r\n\r\naudio_content = AudioContent.from_audio_file(path=\"...\")<\/code><\/pre>\n
    Finally, you can invoke the service to get the transcription:<\/span><\/p>\n
    user_input = await audio_to_text_service.get_text_content(audio_content)\r\nprint(user_input)<\/code><\/pre>\n
    To further create an interactive chat app that takes audio as input, please read this blog post<\/a> or see the sample app<\/a> in our GitHub repository.<\/p>\n
    Text-To-Audio<\/span><\/h3>\n
    The second feature we are introducing is the ability to create audio from text. The foundational class for this feature is TextToAudioClientBase<\/code><\/strong>, which includes two public methods:\u00a0get_audio_contents<\/code>\u00a0<\/strong>and\u00a0get_audio_content<\/code><\/strong>. Similar to AudioToTextCientBase<\/code>,t<\/strong>he former returns a list of possible audio generations based on the number requested, while the latter returns a single audio generation. The audio generations are returned as AudioContent<\/code> <\/strong>objects that contain the audio data. To listen to the audio data, AudioContent<\/code>\u00a0<\/strong>provide another convenient method to save the data to an audio file:<\/span><\/p>\n
    audio_content = ...\r\naudio_content.write_to_file(path=PATH_TO_FILE)<\/code><\/pre>\n
    As of the publication of this blog post, the available services include\u00a0OpenAIAudioToText<\/code>\u00a0<\/strong>and\u00a0AzureAudioToText<\/code><\/strong>, allowing you to utilize either your OpenAI endpoints or Azure deployments.<\/span><\/p>\n
    Text-to-Audio Example Using Azure OpenAI<\/span><\/h4>\n
    Please ensure that Semantic Kernel is updated to the latest version. To process audio output, the following components are required:<\/span><\/p>\n
      \n
    1. A text-to-speech model, such as tts<\/span><\/li>\n
    2. An audio output device<\/span><\/li>\n<\/ol>\n
      To begin, create the service with the following code:<\/span><\/p>\n
      \n
      \n
      from semantic_kernel.connectors.ai.open_ai.services.azure_text_to_audio import AzureTextToAudio\r\n\r\n\r\ntext_to_audio_service = AzureTextToAudio(api_key=\"...\", deployment_name=\"...\", endpoint=\"...\")<\/code><\/pre>\n<\/div>\n<\/div>\n
      Next<\/span>, you can invoke the service to get an audio generation:<\/span><\/p>\n
      audio_content = await text_to_audio_service.get_audio_content(\"Hello World!\")<\/code><\/pre>\n
      Finally, save the audio content so that you can listen to it with your favorite player:<\/span><\/p>\n
      audio_content.write_to_file(path=\"...\")<\/code><\/pre>\n
      To further create an interactive chat app that output audio, please read more in this blog post<\/a> or see the sample app<\/a> in our GitHub repository.<\/p>\n
      Conclusion