\n The screenshot below shows the two new buttons that enable this feature:\n<\/p>\n
- \n
- Microphone <\/strong>\u2013 press to start listening and then speak your query\n <\/li>\n
- Speaker-mute<\/strong> \u2013 when the app is speaking the response, press this button to stop.\n <\/li>\n<\/ul>\n
\n
![\"Image](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/09\/a-screenshot-of-a-computer-description-automatica-1.png\")
\n The speech that is transcribed will be added to the chat as though it was typed and sent directly to the LLM. The LLM\u2019s response is then automatically spoken back through the speakers\/headset. Both the speech-in and speech-out functionality will use built-in Android APIs.\n<\/p>\n
Speech in<\/h2>\n
\n To listen to the user\u2019s question and create a prompt for the LLM, we\u2019re going to use the Android
SpeechRecognizer<\/code><\/a> API. We don\u2019t want the phone to be in permanent listen-mode, so the user will have to tap the microphone icon before speaking. This requires us to:\n<\/p>\n- \n
- \n Configure permissions in AndroidManifest.xml<\/strong>\n <\/li>\n
- \n Check\/ask for permission in code\n <\/li>\n
- \n Initialize API with context\n <\/li>\n
- \n Add API methods to listen for speech and send text to the view model\n <\/li>\n
- \n Wire up UI button to start listening\n <\/li>\n<\/ul>\n
Set & check permissions<\/h3>\n
\n In order for the app to access the microphone, it must request the
RECORD_AUDIO<\/code> permission in AndroidManifest.xml<\/strong>:\n<\/p>\n<uses-permission android:name=\"android.permission.RECORD_AUDIO\"\/><\/pre>\n
\n and then call this
checkPermission<\/code> function in theNavActivity.onCreate<\/code> function:\n<\/p>\nprivate fun checkPermission() {\r\n if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) {\r\n ActivityCompat.requestPermissions(\r\n this,\r\n arrayOf<String>(Manifest.permission.RECORD_AUDIO), RecordAudioRequestCode\r\n )\r\n }\r\n}<\/pre>\n\n Assuming the user agrees, the app will be able to listen to their spoken questions.\n<\/p>\n
Add listening code<\/h3>\n
\n On the
ChannelNameBar<\/code> composable, there\u2019s a new parameteronListenPressed<\/code>, which is called when the icon is clicked (see the \u201cWire up\u2026\u201d section below). This function is implemented in theConversationFragment<\/code> and passed through theConversationContent<\/code> composable. \n<\/p>\n\n The
speechToText<\/code> object is initialized as part of the fragment\u2019sonCreate<\/code> function and takes a number of parameters including setting the expected language. This is so the API can access thecontext<\/code> required for its constructor:\n<\/p>\nspeechToText = SpeechRecognizer.createSpeechRecognizer(this.context)\r\nspeechToText.setRecognitionListener(this)\r\nrecognizerIntent = Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH)\r\nrecognizerIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_PREFERENCE, \"US-en\")<\/pre>\n
\n Once that\u2019s initialized, the
listen<\/code> function gets called via the view model and starts listening for speech to transcribe: \n<\/p>\nspeechToText.startListening(recognizerIntent)<\/pre>\n
\n The rest of the functions required to support the API are also implemented in the fragment. When some text is successfully transcribed, the result is passed to the view model via
setSpeech<\/code>.\n<\/p>\n\n The
MainViewModel.setSpeech<\/code> function inserts the text into the existing \u2018workflow\u2019 for new messages, so it gets added in the user interface and the existing code is triggered to send to the LLM.\n<\/p>\nfun setSpeech (text: String) {\r\n onMessageSent(text)\r\n}<\/pre>\nOnce
onMessageSent<\/code> is called, the app behaves as if the query was input via the keyboard.<\/strong>\n<\/p>\nSpeech out<\/h2>\n
\n To read the model\u2019s responses aloud, we\u2019ve used the Android
TextToSpeech<\/code><\/a> API. Like theSpeechToText<\/code> class, it gets initialized in the fragment\u2019sonCreate<\/code> function:\n<\/p>\ntts = TextToSpeech(this.context, this) \/\/ context, fragment\r\nactivityViewModel.setSpeechGenerator(tts)<\/pre>\n
\n Notice the second constructor parameter is a reference to the fragment itself \u2013 this is because we\u2019ve implemented the
TextToSpeech.OnInitListener<\/code> interface on the fragment. The mute button state (disabled except when there is audio generated) is based on those callback functions.\n<\/p>\n\n In the
onMessageSent<\/code> function, after a response has been received from theOpenAiWrapper<\/code> and displayed in the UI, the code will also<\/em> read it aloud<\/strong>:\n<\/p>\ntextToSpeech.speak(chatResponse, TextToSpeech.QUEUE_FLUSH, null,\"\")<\/pre>\n
\n As with the speech recognition implementation, there is minimal change to the existing logic of the app.\n<\/p>\n
Access the Android context<\/h2>\n
\n As mentioned above, both APIs need to be configured with a
Context<\/code>, so they are created and initialized in theConversationFragment<\/code> with references set in theMainViewModel<\/code>. TheMainViewModel<\/code> encapsulates the speech-related APIs and exposes methods that are called from the Jetpack Compose UI.\n<\/p>\n\n The other reason for implementing the functionality on the fragment is that we need to implement the interfaces
RecognitionListener<\/code> andTextToSpeech.OnInitListener<\/code>, for instance to capture and update the mutable statespeechState<\/code>.\n<\/p>\nNOTE: you will also see a method
setContext<\/code> on theMainViewModel<\/code> – this is used by the Sqlite implementations for vector caching and history embedding \u2013 and is not related to the speech features.<\/p><\/blockquote>\nWire up the Jetpack Compose UI<\/h2>\n
\n The above snippets show how the speech recognition and text-to-speech APIs are wired up. The next code snippet shows how the user interface triggers the functionality from the view model via composables in Conversation.kt<\/strong>. The speech UI controls (record and mute) are added as icons to the
ChannelNameBar<\/code> composable. In the composable where the icons are declared you can see these functions and state are used to trigger events on the view model:\n<\/p>\n- \n
- \n
onListenPressed()<\/code>\n <\/li>\n- \n
speechState<\/code>\n <\/li>\n- \n
onStopTalkingPressed()<\/code>\n <\/li>\n<\/ul>\n\/\/ \"Microphone\" icon\r\nIconButton(onClick = {\r\n onListenPressed()\r\n}) {\r\n Icon(\r\n imageVector = when (speechState) {\r\n SpeechState.LISTENING -> Icons.Filled.KeyboardVoice\r\n else -> Icons.Outlined.KeyboardVoice\r\n },\r\n tint = when (speechState) {\r\n SpeechState.LISTENING -> MaterialTheme.colorScheme.primary\r\n else -> MaterialTheme.colorScheme.onSurfaceVariant\r\n },\r\n modifier = Modifier.clip(CircleShape),\r\n contentDescription = stringResource(id = R.string.enable_mic)\r\n )\r\n}\r\n\/\/ \"End speaking\" icon\r\nIconButton(\r\n onClick = {\r\n onStopTalkingPressed()\r\n },\r\n enabled = when (speechState) {\r\n SpeechState.SPEAKING -> true\r\n else -> false\r\n }\r\n ) {\r\n Icon(\r\n imageVector = Icons.Outlined.VolumeOff,\r\n modifier = Modifier.clip(CircleShape),\r\n contentDescription = stringResource(id = R.string.mute_tts)\r\n )\r\n}<\/pre>\n\n These are set in the
ConversationContent<\/code> composable and implemented in theConversationFragment<\/code>, calling into functions or mutable state (eg. thespeechState<\/code>) defined in the fragment.\n<\/p>\nOne last thing\u2026<\/h2>\n
\n After adding this feature that reads out every response, it started driving me crazy while testing other features. I added a flag
Constants.ENABLE_SPEECH<\/code> that should be set totrue<\/code> to use these features, butfalse<\/code> when you are testing or otherwise don\u2019t wish to utilize theSpeechRecognition<\/code> andTextToSpeech<\/code> functionality. This could be an application preference in future.\n<\/p>\n\n If you\u2019re running the sample and find the speech features not working, check that
ENABLE_SPEECH = true<\/code> in Constants.kt<\/strong> and that you agreed to the permissions dialog when the app started.\n<\/p>\nResources and feedback<\/h2>\n
\n See the Azure OpenAI documentation<\/a> for more information on the wide variety of services available for your apps.\n<\/p>\n
\n We\u2019d love your feedback on this post, including any tips or tricks you\u2019ve learned from playing around with ChatGPT prompts.\n<\/p>\n
\n If you have any thoughts or questions, use the feedback forum<\/a> or message us on Twitter @surfaceduodev<\/a>.\n<\/p>\n
- \n
- Speaker-mute<\/strong> \u2013 when the app is speaking the response, press this button to stop.\n <\/li>\n<\/ul>\n