{"id":3680,"date":"2024-11-15T08:54:18","date_gmt":"2024-11-15T16:54:18","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/semantic-kernel\/?p=3680"},"modified":"2024-11-15T08:54:18","modified_gmt":"2024-11-15T16:54:18","slug":"allow-users-to-talk-and-listen-to-your-chatbot-using-semantic-kernel-python","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/agent-framework\/allow-users-to-talk-and-listen-to-your-chatbot-using-semantic-kernel-python\/","title":{"rendered":"Allow users to talk and listen to your chatbot using Semantic Kernel Python"},"content":{"rendered":"

Until now, Semantic Kernel Python only allowed for the development of text-based AI applications. However, this is no longer the case, as we have expanded its capabilities to include audio as one of the supported modalities. In this article, I will provide a detailed, step-by-step guide on how to create a chatbot that can both speak to and listen to your users.<\/span><\/p>\n

As of the time of this blog post, OpenAI has announced the release of the Realtime API. For further details, you can find more information here<\/a>. Please note that this blog post is not intended as a tutorial on the Realtime API. The Semantic Kernel team remains committed to delivering the latest advancements in AI to all developers. We encourage you to stay tuned for future updates.<\/em><\/span><\/p>\n

Step 1: Create a chatbot<\/span><\/h3>\n
    \n
  1. Please make sure you have the latest Semantic Kernel Python installed.<\/span><\/li>\n
  2. Please make sure you have an Azure OpenAI chat completion model deployment or an OpenAI endpoint.<\/span><\/li>\n<\/ol>\n
    # Copyright (c) Microsoft. All rights reserved.\r\n\r\nimport asyncio\r\nimport logging\r\nimport os\r\n\r\nfrom semantic_kernel.connectors.ai.open_ai import (\r\n    AzureChatCompletion,\r\n    OpenAIChatPromptExecutionSettings,\r\n)\r\nfrom semantic_kernel.contents import ChatHistory\r\n\r\nlogging.basicConfig(level=logging.WARNING)\r\n\r\nsystem_message = \"\"\"\r\nYou are a chat bot. Your name is Mosscap and\r\nyou have one goal: figure out what people need.\r\nYour full name, should you need to know it, is\r\nSplendid Speckled Mosscap. You communicate\r\neffectively, but you tend to answer with long\r\nflowery prose.\r\n\"\"\"\r\n\r\n\r\nchat_service = AzureChatCompletion()\r\n\r\nhistory = ChatHistory(system_message=system_message)\r\nhistory.add_user_message(\"Hi there, who are you?\")\r\nhistory.add_assistant_message(\"I am Mosscap, a chat bot. I'm trying to figure out what people need.\")\r\n\r\n\r\nasync def chat() -> bool:\r\n    try:\r\n        user_input = input(\"User:> \")\r\n    except KeyboardInterrupt:\r\n        print(\"\\n\\nExiting chat...\")\r\n        return False\r\n    except EOFError:\r\n        print(\"\\n\\nExiting chat...\")\r\n        return False\r\n\r\n    if \"exit\" in user_input.lower():\r\n        print(\"\\n\\nExiting chat...\")\r\n        return False\r\n\r\n    history.add_user_message(user_input)\r\n\r\n    chunks = chat_service.get_streaming_chat_message_content(\r\n        chat_history=history,\r\n        settings=OpenAIChatPromptExecutionSettings(\r\n            max_tokens=2000,\r\n            temperature=0.7,\r\n            top_p=0.8,\r\n        ),\r\n    )\r\n\r\n    print(\"Mosscap:> \", end=\"\")\r\n    answer = \"\"\r\n    async for message in chunks:\r\n        print(str(message), end=\"\")\r\n        answer += str(message)\r\n    print(\"\\n\")\r\n\r\n    history.add_assistant_message(str(answer))\r\n\r\n    return True\r\n\r\n\r\nasync def main() -> None:\r\n    chatting = True\r\n    while chatting:\r\n        chatting = await chat()\r\n\r\n\r\nif __name__ == \"__main__\":\r\n    asyncio.run(main())\r\n<\/code><\/pre>\n
    In the code snippet provided, we begin by defining a system message that establishes the personality of the chatbot. Subsequently, we create a chat completion service utilizing the Azure OpenAI connector, along with a chat history containing pre-populated messages to initiate the conversation. Finally, we implement a loop that captures user input and generates a streaming response. It is important to note that both user inputs and model responses will be stored in the chat history, allowing the chatbot to maintain the context of the conversation throughout each iteration.<\/span><\/p>\n
    Step 2: Allow the chatbot to listen to you<\/span><\/h3>\n
      \n
    1. Please make sure you have an Azure OpenAI speech-to-text model (i.e. whisper) deployment or an OpenAI endpoint.\u00a0<\/span><\/li>\n
    2. Python dependency: pyaudio<\/strong><\/em> for working with audio<\/li>\n
    3. Python dependency: keyboard <\/strong><\/em>for controlling audio input duration<\/li>\n<\/ol>\n
      pip install pyaudio\r\npip install keyboard<\/code><\/pre>\n
      Our goal is to convert audio into text, a step commonly referred to as “transcription”.<\/span><\/p>\n
      from semantic_kernel.connectors.ai.open_ai import AzureAudioToText\r\n\r\naudio_to_text_service = AzureAudioToText()\r\n<\/code><\/pre>\n
      We first create an audio-to-text service. We are doing it here with the AzureAudioToText<\/code> <\/strong>connector.\u00a0<\/span><\/p>\n
      # Copyright (c) Microsoft. All rights reserved.\r\n\r\nimport os\r\nimport wave\r\nfrom typing import ClassVar\r\n\r\nimport keyboard\r\nimport pyaudio\r\nfrom pydantic import BaseModel\r\n\r\n\r\nclass AudioRecorder(BaseModel):\r\n    \"\"\"A class to record audio from the microphone and save it to a WAV file.\r\n\r\n    To start recording, press the spacebar. To stop recording, release the spacebar.\r\n\r\n    To use as a context manager, that automatically removes the output file after exiting the context:\r\n    ```\r\n    with AudioRecorder(output_filepath=\"output.wav\") as recorder:\r\n        recorder.start_recording()\r\n        # Do something with the recorded audio\r\n        ...\r\n    ```\r\n    \"\"\"\r\n\r\n    # Audio recording parameters\r\n    FORMAT: ClassVar[int] = pyaudio.paInt16\r\n    CHANNELS: ClassVar[int] = 1\r\n    RATE: ClassVar[int] = 44100\r\n    CHUNK: ClassVar[int] = 1024\r\n\r\n    output_filepath: str\r\n\r\n    def start_recording(self) -> None:\r\n        # Wait for the spacebar to be pressed to start recording\r\n        keyboard.wait(\"space\")\r\n\r\n        # Start recording\r\n        audio = pyaudio.PyAudio()\r\n        stream = audio.open(\r\n            format=self.FORMAT,\r\n            channels=self.CHANNELS,\r\n            rate=self.RATE,\r\n            input=True,\r\n            frames_per_buffer=self.CHUNK,\r\n        )\r\n        frames = []\r\n\r\n        while keyboard.is_pressed(\"space\"):\r\n            data = stream.read(self.CHUNK)\r\n            frames.append(data)\r\n\r\n        # Recording stopped as the spacebar is released\r\n        stream.stop_stream()\r\n        stream.close()\r\n\r\n        # Save the recorded data as a WAV file\r\n        with wave.open(self.output_filepath, \"wb\") as wf:\r\n            wf.setnchannels(self.CHANNELS)\r\n            wf.setsampwidth(audio.get_sample_size(self.FORMAT))\r\n            wf.setframerate(self.RATE)\r\n            wf.writeframes(b\"\".join(frames))\r\n\r\n        audio.terminate()\r\n\r\n    def remove_output_file(self) -> None:\r\n        os.remove(self.output_filepath)\r\n\r\n    def __enter__(self) -> \"AudioRecorder\":\r\n        return self\r\n\r\n    def __exit__(self, exc_type, exc_value, traceback) -> None:\r\n        self.remove_output_file()<\/code><\/pre>\n
      Next, we will create a helper class named AudioRecorder<\/code><\/strong>, which facilitates easier interaction with audio functionality on your system. This class initiates recording when the user presses and holds the space bar on the keyboard, and it ceases recording upon the release of the key. The recorded audio is saved as a file on the disk. Additionally, when used as a context manager, the audio file will be automatically deleted after the audio processing is completed.<\/span><\/p>\n
      AUDIO_FILEPATH = os.path.join(os.path.dirname(__file__), \"output.wav\")\r\n\r\ntry:\r\n    print(\"User:> \", end=\"\", flush=True)\r\n    with AudioRecorder(output_filepath=AUDIO_FILEPATH) as recorder:\r\n        recorder.start_recording()\r\n        user_input = await audio_to_text_service.get_text_content(AudioContent.from_audio_file(AUDIO_FILEPATH))\r\n        print(user_input.text)\r\nexcept KeyboardInterrupt:\r\n    print(\"\\n\\nExiting chat...\")\r\n    return False\r\nexcept EOFError:\r\n    print(\"\\n\\nExiting chat...\")\r\n    return False\r\n\r\nif \"exit\" in user_input.text.lower():\r\n    print(\"\\n\\nExiting chat...\")\r\n    return False\r\n\r\nhistory.add_user_message(user_input.text)<\/code><\/pre>\n
      Finally, we utilize the AudioRecorder<\/code><\/strong>to capture user input and subsequently transcribe the audio into text for the chat completion service. It is important to note that, since the audio-to-text service returns a TextContent<\/code><\/strong>object, we must retrieve the text by accessing itstext<\/strong><\/code>property.<\/span><\/p>\n
      \n
      \n
      \n
      \n
      \n
      \n
      \n
      \n
      \n
      \n
      \n
      \n
      \n
      To view the complete sample, please visit this link<\/a> to our GitHub repository.<\/span><\/em><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
      Now, please run the application. Hold down the space bar on your keyboard and begin speaking. Once you have finished, release the key and wait for the response to be displayed on the screen.<\/span><\/p>\n
      Step 3: Allow the chatbot to talk to you<\/span><\/h3>\n
      What we have achieved thus far is promising, but it is not yet complete. It is time to incorporate the final component.<\/p>\n
        \n
      1. Please make sure you have an Azure OpenAI text-to-speech model (i.e. tts) deployment or an OpenAI endpoint.\u00a0<\/span><\/li>\n<\/ol>\n
        Our goal is to convert the response generated by the chat completion service back to audio.<\/span><\/p>\n
        from semantic_kernel.connectors.ai.open_ai import AzureTextToAudio\r\n\r\ntext_to_audio_service = AzureTextToAudio()\r\n<\/code><\/pre>\n
        We first create a text-to-audio service. We are doing it here with the AzureTextToAudio<\/code><\/strong>connector.<\/span><\/p>\n
        audio_content = await text_to_audio_service.get_audio_content(\r\n    response.content, OpenAITextToAudioExecutionSettings(response_format=\"wav\")\r\n)<\/code><\/pre>\n
        Next, we will invoke the text-to-audio service to generate audio for the response. At this stage, we are specifying the output format for reasons that will be addressed later. With the OpenAITextToAudioExecutionSettings<\/code>, you can also define the type of voice and the speed of the audio. Please feel free to experiment with different settings to discover those that you find most comfortable.<\/span><\/p>\n
        # Copyright (c) Microsoft. All rights reserved.\r\n\r\nimport io\r\nimport logging\r\nimport wave\r\nfrom typing import ClassVar\r\n\r\nimport pyaudio\r\nfrom pydantic import BaseModel\r\n\r\nfrom semantic_kernel.contents import AudioContent\r\n\r\nlogging.basicConfig(level=logging.WARNING)\r\nlogger: logging.Logger = logging.getLogger(__name__)\r\n\r\n\r\nclass AudioPlayer(BaseModel):\r\n    \"\"\"A class to play an audio file to the default audio output device.\"\"\"\r\n\r\n    # Audio replay parameters\r\n    CHUNK: ClassVar[int] = 1024\r\n\r\n    audio_content: AudioContent\r\n\r\n    def play(self, text: str | None = None) -> None:\r\n        \"\"\"Play the audio content to the default audio output device.\r\n\r\n        Args:\r\n            text (str, optional): The text to display while playing the audio. Defaults to None.\r\n        \"\"\"\r\n        audio_stream = io.BytesIO(self.audio_content.data)\r\n        with wave.open(audio_stream, \"rb\") as wf:\r\n            audio = pyaudio.PyAudio()\r\n            stream = audio.open(\r\n                format=audio.get_format_from_width(wf.getsampwidth()),\r\n                channels=wf.getnchannels(),\r\n                rate=wf.getframerate(),\r\n                output=True,\r\n            )\r\n\r\n            if text:\r\n                # Simulate the output of text while playing the audio\r\n                data_frames = []\r\n\r\n                data = wf.readframes(self.CHUNK)\r\n                while data:\r\n                    data_frames.append(data)\r\n                    data = wf.readframes(self.CHUNK)\r\n\r\n                if len(data_frames) < len(text):\r\n                    logger.warning(\r\n                        \"The audio is too short to play the entire text. \",\r\n                        \"The text will be displayed without synchronization.\",\r\n                    )\r\n                    print(text)\r\n                else:\r\n                    for data_frame, text_frame in self._zip_text_and_audio(text, data_frames):\r\n                        stream.write(data_frame)\r\n                        print(text_frame, end=\"\", flush=True)\r\n                    print()\r\n            else:\r\n                data = wf.readframes(self.CHUNK)\r\n                while data:\r\n                    stream.write(data)\r\n                    data = wf.readframes(self.CHUNK)\r\n\r\n            stream.stop_stream()\r\n            stream.close()\r\n            audio.terminate()\r\n\r\n    def _zip_text_and_audio(self, text: str, audio_frames: list) -> zip:\r\n        \"\"\"Zip the text and audio frames together so that they can be displayed in sync.\r\n\r\n        This is done by evenly distributing empty strings between each character and\r\n        append the remaining empty strings at the end.\r\n\r\n        Args:\r\n            text (str): The text to display while playing the audio.\r\n            audio_frames (list): The audio frames to play.\r\n\r\n        Returns:\r\n            zip: The zipped text and audio frames.\r\n        \"\"\"\r\n        text_frames = list(text)\r\n        empty_string_count = len(audio_frames) - len(text_frames)\r\n        empty_string_spacing = len(text_frames) \/\/ empty_string_count\r\n\r\n        modified_text_frames = []\r\n        current_empty_string_count = 0\r\n        for i, text_frame in enumerate(text_frames):\r\n            modified_text_frames.append(text_frame)\r\n            if current_empty_string_count < empty_string_count and i % empty_string_spacing == 0:\r\n                modified_text_frames.append(\"\")\r\n                current_empty_string_count += 1\r\n\r\n        if current_empty_string_count < empty_string_count:\r\n            modified_text_frames.extend([\"\"] * (empty_string_count - current_empty_string_count))\r\n\r\n        return zip(audio_frames, modified_text_frames)<\/code><\/pre>\n
        Now, we have the audio; however, we currently lack a method to play it for the user. To address this, we are creating another helper class called AudioPlayer<\/code>, which will accept an AudioContent<\/code>and play the audio through your speakers. This helper class will also synchronize the audio and the text when provided, creating a streaming effect.<\/span><\/p>\n
        print(\"Mosscap:> \", end=\"\", flush=True)\r\nAudioPlayer(audio_content=audio_content).play(text=response.content)<\/code><\/pre>\n
        Finally, we play the audio and display the response on the screen.<\/span><\/p>\n
        \n
        \n
        \n
        \n
        \n
        \n
        \n
        \n
        \n
        \n
        \n
        \n
        \n
        To view the complete sample, please visit this link<\/a> to our GitHub repository.<\/span><\/em><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
        Now, please run the application. Press and hold the space bar on your keyboard while you begin speaking. Once you have finished, release the key and wait for the response to be spoken to you.<\/span><\/p>\n
        Conclusion<\/span><\/h3>\n
        In this blog post, we show you how to incorporate audio into your AI application to elevate its experience with just a few lines of code. In fact, it takes even more code to record and play audio on your computer! To learn more about the basics, you can read more in this blog post<\/a> or visit our learn site<\/a> as well as our GitHub repository<\/a>.<\/p>\n
         <\/p>\n
        \n
        \n
        \n
        \n
        The Semantic Kernel team is dedicated to empowering developers by providing access to the latest advancements in the industry. We encourage you to leverage your creativity and build remarkable solutions with SK! Please reach out if you have any questions or feedback through our\u00a0Semantic Kernel GitHub Discussion Channel<\/a>. We look forward to hearing from you!\u00a0We would also love your support, if you\u2019ve enjoyed using Semantic Kernel, give us a star on\u00a0GitHub<\/a>.<\/em><\/span><\/p>\n<\/div>\n<\/article>\n<\/div>\n<\/article>\n","protected":false},"excerpt":{"rendered":"
        Until now, Semantic Kernel Python only allowed for the development of text-based AI applications. However, this is no longer the case, as we have expanded its capabilities to include audio as one of the supported modalities. In this article, I will provide a detailed, step-by-step guide on how to create a chatbot that can both […]<\/p>\n","protected":false},"author":165150,"featured_media":2311,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[34,1],"tags":[48,63,9],"class_list":["post-3680","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-python-2","category-semantic-kernel","tag-ai","tag-microsoft-semantic-kernel","tag-semantic-kernel"],"acf":[],"blog_post_summary":"
        Until now, Semantic Kernel Python only allowed for the development of text-based AI applications. However, this is no longer the case, as we have expanded its capabilities to include audio as one of the supported modalities. In this article, I will provide a detailed, step-by-step guide on how to create a chatbot that can both […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/3680","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/users\/165150"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/comments?post=3680"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/posts\/3680\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media\/2311"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/media?parent=3680"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/categories?post=3680"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/agent-framework\/wp-json\/wp\/v2\/tags?post=3680"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}