Optimizing Speech-to-Text (STT) and Text-to-Speech (TTS) is essential for developers building voice-enabled applications, as it enhances recognition accuracy and overall user experience. Additionally, custom keyword recognition enables hands-free activation for voice assistants.<\/p>\n
This blog outlines:<\/p>\n
Azure provides several STT models:<\/p>\n
Azure Speech-to-Text (STT) supports a wide range of languages and locales using default models. You can find the complete list of supported languages and features in the official documentation: Supported languages for Speech-to-Text<\/a>.<\/p>\n To improve speech to text recognition accuracy, customization is available for some languages and base models. Depending on the locale, you can upload audio + human-labeled transcripts, plain text, structured text, and pronunciation data. By default, plain text customization<\/strong> is supported for all available base models<\/a>.<\/p>\n The following Embedded STT models are currently available:<\/p>\n You can check the official documentation for the most up-to-date list of supported languages and models:\nEmbedded Speech Models and Voices<\/a>.<\/p>\n Several techniques enhance STT accuracy by correcting misrecognitions and adapting the model to domain-specific vocabulary.<\/p>\n A phrase list improves recognition for specific words and phrases. This is particularly useful for proper nouns, brand names, and technical terms. For more details, refer to the Improve recognition accuracy with phrase list<\/a>.<\/p>\n Phrase lists alone may not be sufficient for certain words. Custom logic can be implemented to force corrections of frequently misrecognized words.<\/p>\n To refine the correction logic, two methods can be used to identify misrecognition patterns:<\/p>\n N-best refers to a ranked list of recognition hypotheses, where the STT engine provides not only the top result but also alternative candidates. By analyzing lower-ranked alternatives, it is possible to identify and correct systematic misrecognitions.<\/p>\n To retrieve N-best results and detailed word-level recognition information, configure Once the STT engine provides N-best results, the application can analyze them to refine recognition accuracy. Below is an example of how to parse N-best results from the JSON response:<\/p>\n When an entire sentence is processed instead of a single word, N-best alternatives focus on specific words within the sentence. Below is an actual example where the word “out”<\/strong> has multiple alternatives:<\/p>\n In this example, “toon” is an alternative candidate for “tune” (SourceSpan = 0), making it a possible correction in the phrase “tune goggles.”<\/p>\n This insight helps refine correction rules, ensuring misrecognitions like “tune goggles” \u2192 “ToonGoggles” are automatically corrected. By analyzing lower-confidence alternatives, developers can improve STT accuracy for domain-specific terms.<\/p>\n To efficiently apply correction rules, a dedicated correction utility class ( To avoid case-sensitivity issues, all entries in Azure provides several TTS models:<\/p>\n Cloud Neural Voices supports 150+ languages and accents<\/strong> with 500+ voice models<\/strong>. You can find the complete list of supported languages and features in the official documentation: Supported languages for Text-to-Speech<\/a>.<\/p>\n CNV includes several feature types\u2014CNV Pro, CNV Lite, Cross-lingual voice source and target, and Multi-style voice\u2014and the supported locales vary depending on the feature selected. For more details, refer to the Supported languages for Custom Neural Voice<\/a>.<\/p>\n For comparison with Speech-to-Text (STT), all Text-to-Speech (TTS) locales here<\/a> (except Persian (fa-IR)) are available out of the box with at least one selected female and\/or male voice per locale<\/strong>. If a required voice model is not available in Embedded Neural Voices, additional cost may be required<\/strong> to train a new model.<\/p>\n To correct mispronunciations, a custom lexicon can be created using SSML<\/strong> or an external XML-based lexicon file<\/strong>. For more details, refer to the Custom Lexicon documentation<\/a>.<\/p>\n SSML supports inline phoneme correction using For more details, refer to:\nSSML Pronunciation Guide<\/a><\/p>\n For multiple custom pronunciations, an XML-based pronunciation lexicon file<\/strong> can be used.\nThis file should follow the Pronunciation Lexicon Specification (PLS) Version 1.0<\/strong>.<\/p>\n Supported Phonetic Alphabets:<\/strong><\/p>\n For more details: Custom Lexicon Documentation<\/a><\/p>\n The custom lexicon should be stored as To reference the custom lexicon in SSML, use the Important Notes:<\/strong><\/p>\n Microsoft provides a validation tool to ensure that custom lexicons are correctly formatted:\nCustom Lexicon Validation Tool<\/a><\/p>\n Developers can train custom keywords using the Custom Keyword portal in Azure AI Speech Studio<\/strong>. This is a self-serve experience that allows customers to create models without requiring Microsoft’s intervention.<\/p>\nCustom Speech Models<\/h4>\n
Embedded Speech Models<\/h4>\n
\n
Improving Recognition Accuracy<\/h3>\n
Phrase List (For both Embedded and Cloud models)<\/h4>\n
Implementation Example (Java)<\/h5>\n
import com.microsoft.cognitiveservices.speech.PhraseListGrammar;\r\n\r\nPhraseListGrammar phraseList = PhraseListGrammar.fromRecognizer(recognizer);\r\nphraseList.addPhrase(\"Microsoft\");\r\nphraseList.addPhrases([\"Azure\", \"Teams\"]);<\/code><\/pre>\nAdditional Features<\/h5>\n
\n
phraseList.clear();<\/code><\/pre>\n<\/li>\nCustom Correction Logic (For both Embedded and Cloud models (only en-US<\/strong>))<\/h4>\n
Preparing
corrections.json<\/code><\/h5>\n\n
What is N-best?<\/h5>\n
Enabling N-best in Word Level Details<\/h5>\n
speechConfig<\/code> with the following properties:<\/p>\nspeechConfig.setSpeechRecognitionOutputFormat(OutputFormat.Detailed);\r\nspeechConfig.setProperty(\"SpeechRecognition_RequestWordLevelCorrections\", \"true\");<\/code><\/pre>\nParsing N-best Results<\/h5>\n
\/\/ Fetch JSON result containing N-best\r\nString jsonResult = e.getResult()\r\n .getProperties()\r\n .getProperty(PropertyId.SpeechServiceResponse_JsonResult);\r\nSystem.out.println(\"JSON result: \" + jsonResult);\r\n\r\n\/\/ Parse JSON for N-best recognition candidates\r\nJSONObject json = new JSONObject(jsonResult);\r\nJSONArray nbestArray = json.getJSONArray(\"NBest\");\r\n\r\nif (nbestArray != null && nbestArray.length() > 0) {\r\n for (int i = 0; i < nbestArray.length(); i++) {\r\n JSONObject candidate = nbestArray.getJSONObject(i);\r\n System.out.println(\"Candidate \" + (i+1) + \": \" + candidate.getString(\"Display\") +\r\n \" (Confidence: \" + candidate.getDouble(\"Confidence\") + \")\");\r\n }\r\n}\r\n<\/code><\/pre>\nN-best Example with Sentence Input<\/h5>\n
\"NBest\": [\r\n {\r\n \"Confidence\": 0.867123,\r\n \"Lexical\": \"tune goggles\",\r\n \"Display\": \"Tune Goggles.\",\r\n \"Words\": [\r\n {\r\n \"Word\": \"tune\",\r\n \"Confidence\": 0.867123,\r\n \"Offset\": 12500000,\r\n \"Duration\": 4500000\r\n },\r\n {\r\n \"Word\": \"goggles\",\r\n \"Confidence\": 0.867123,\r\n \"Offset\": 17500000,\r\n \"Duration\": 7000000\r\n }\r\n ]\r\n ...\r\n }\r\n],\r\n\"Corrections\": {\r\n \"CorrectionCandidates\": [\r\n {\r\n \"Alternates\": [\r\n { \"AlternateWords\": [\"toon\"], \"Id\": 0, \"SourceSpan\": [0] }\r\n ],\r\n \"Confidence\": \"High\",\r\n \"Id\": 0,\r\n \"Span\": [0]\r\n }\r\n ],\r\n}<\/code><\/pre>\nExample Correction Rules (
corrections.json<\/code>)<\/h5>\n{\r\n \"ToonGoggles\": [\"tune goggles\", \"toon googles\", \"tune google\"]\r\n}<\/code><\/pre>\nImplementing Correction Logic<\/h4>\n
CorrectionUtils<\/code>) is used. This class loads misrecognition patterns from corrections.json<\/code> and automatically corrects recognized text.<\/p>\nKey Features of
CorrectionUtils<\/code><\/h5>\n\n
corrections.json<\/code> without changing application code.<\/li>\n<\/ul>\nCode Snippet of
CorrectionUtils<\/code> (Java)<\/h5>\npublic class CorrectionUtils {\r\n private static final Map<String, List<String>> correctionMap = new HashMap<>();\r\n\r\n static { loadCorrections(\"corrections.json\"); }\r\n\r\n private static void loadCorrections(String filePath) {\r\n try {\r\n \/\/ Load JSON file\r\n ObjectMapper mapper = new ObjectMapper();\r\n Map<String, List<String>> corrections = mapper.readValue(\r\n new BufferedReader(new FileReader(filePath)),\r\n new TypeReference<Map<String, List<String>>>() {}\r\n );\r\n\r\n \/\/ Convert all misrecognition lists to lowercase\r\n for (Map.Entry<String, List<String>> entry : corrections.entrySet()) {\r\n correctionMap.put(entry.getKey(),\r\n entry.getValue().stream()\r\n .map(String::toLowerCase)\r\n .collect(Collectors.toList())\r\n );\r\n }\r\n } catch (IOException e) {\r\n e.printStackTrace();\r\n }\r\n }\r\n\r\n public static String correctRecognition(String recognizedText) {\r\n String lowerCaseText = recognizedText.toLowerCase(); \/\/ Convert input to lowercase\r\n for (Map.Entry<String, List<String>> entry : correctionMap.entrySet()) {\r\n String correctPhrase = entry.getKey();\r\n List<String> misrecognitions = entry.getValue();\r\n\r\n for (String misrecognition : misrecognitions) {\r\n if (lowerCaseText.contains(misrecognition)) {\r\n \/\/ Correct the recognition result\r\n recognizedText = recognizedText.replaceAll(\"(?i)\" + misrecognition, correctPhrase);\r\n }\r\n }\r\n }\r\n return recognizedText;\r\n }\r\n}<\/code><\/pre>\ncorrections.json<\/code> should be written in lowercase, ensuring consistent application across different inputs.<\/p>\nUsage Example in Speech Recognition<\/h5>\n
String recognizedText = e.getResult().getText();\r\nrecognizedText = CorrectionUtils.correctRecognition(recognizedText);\r\nSystem.out.println(\"Corrected: Text=\" + recognizedText);<\/code><\/pre>\nText-to-Speech (TTS) Optimization<\/h2>\n
Obtaining TTS Models<\/h3>\n
\n
TTS Locale Support<\/h3>\n
Cloud Neural Voices<\/h4>\n
Custom Neural Voice (CNV)<\/h4>\n
Embedded Neural Voices<\/h4>\n
Custom Lexicon for Pronunciation<\/h3>\n
Defining Pronunciation Using SSML<\/h4>\n
<phoneme><\/code> and <sub><\/code> elements.<\/p>\n\n
<phoneme><\/code> allows pronunciation adjustments using International Phonetic Alphabet (IPA)<\/strong>, Speech API (SAPI)<\/strong>, Universal Phone Set (UPS)<\/strong>, or X-SAMPA<\/strong>.<\/li>\n<sub><\/code> is used for aliasing text.<\/li>\n<\/ul>\nExample<\/h5>\n
<speak version=\"1.0\" xmlns=\"http:\/\/www.w3.org\/2001\/10\/synthesis\"\r\n xmlns:mstts=\"http:\/\/www.w3.org\/2001\/mstts\"\r\n xml:lang=\"en-US\">\r\n <voice name=\"en-US-JennyNeural\">\r\n <phoneme alphabet=\"ipa\" ph=\"\u02c8ma\u026akr\u0259\u02ccs\u0252ft\">Microsoft<\/phoneme>\r\n <sub alias=\"MSFT\">Microsoft<\/sub>\r\n <\/voice>\r\n<\/speak><\/code><\/pre>\nUsing an External Custom Lexicon File<\/h4>\n
Example Custom Lexicon File (PLS XML)<\/h5>\n
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<lexicon version=\"1.0\"\r\n xmlns=\"http:\/\/www.w3.org\/2005\/01\/pronunciation-lexicon\"\r\n alphabet=\"ipa\" xml:lang=\"en-US\">\r\n <lexeme>\r\n <grapheme>Microsoft<\/grapheme>\r\n <phoneme>\u02c8ma\u026akr\u0259\u02ccs\u0252ft<\/phoneme>\r\n <\/lexeme>\r\n <lexeme>\r\n <grapheme>Azure<\/grapheme>\r\n <phoneme>\u02c8\u00e6\u0292\u0259r<\/phoneme>\r\n <\/lexeme>\r\n <lexeme>\r\n <grapheme>Power BI<\/grapheme>\r\n <alias>PowerBI<\/alias>\r\n <\/lexeme>\r\n <lexeme>\r\n <grapheme>PowerBI<\/grapheme>\r\n <phoneme>\u02c8pa\u028a\u0259r\u02ccbi\u02d0\u02cca\u026a<\/phoneme>\r\n <\/lexeme>\r\n <lexeme>\r\n <grapheme>Teams<\/grapheme>\r\n <phoneme>ti\u02d0mz<\/phoneme>\r\n <\/lexeme>\r\n<\/lexicon><\/code><\/pre>\nKey Elements:<\/h5>\n
\n
<lexicon><\/code>: Root element for defining pronunciation rules.<\/li>\n<lexeme><\/code>: Defines pronunciation for each word.<\/li>\n<grapheme><\/code>: Represents the original text.<\/li>\n<phoneme><\/code>: Defines the pronunciation of the word.<\/li>\n<\/ul>\n\n
ipa<\/code>: International Phonetic Alphabet<\/li>\nsapi<\/code>: Microsoft Speech API<\/li>\nups<\/code>: Universal Phone Set<\/li>\nx-sampa<\/code>: Extended SAMPA<\/li>\n<\/ul>\nImplementing Custom Lexicon<\/h3>\n
Saving and Storing the Lexicon File<\/h4>\n
customlexicon.xml<\/code> and must be in UTF-8<\/strong> encoding.\nEmbedded Neural Voices only support offline custom lexicons<\/strong>, so the lexicon file must be stored locally.<\/p>\n\n
Configuring SSML to Use a Custom Lexicon<\/h4>\n
<lexicon><\/code> element with the appropriate file path.<\/p>\nExample SSML for Cloud TTS (Publicly Hosted Lexicon)<\/h5>\n
<speak version=\"1.0\" xmlns=\"http:\/\/www.w3.org\/2001\/10\/synthesis\"\r\n xmlns:mstts=\"http:\/\/www.w3.org\/2001\/mstts\"\r\n xml:lang=\"en-US\">\r\n <voice name=\"en-US-JennyNeural\">\r\n <lexicon uri=\"https:\/\/example.com\/customlexicon.xml\"\/>\r\n Please launch Teams.\r\n <\/voice>\r\n<\/speak><\/code><\/pre>\nExample SSML for Embedded TTS (Local Lexicon)<\/h5>\n
<speak version=\"1.0\" xmlns=\"http:\/\/www.w3.org\/2001\/10\/synthesis\"\r\n xmlns:mstts=\"http:\/\/www.w3.org\/2001\/mstts\"\r\n xml:lang=\"en-US\">\r\n <voice name=\"en-US-JennyNeural\">\r\n <lexicon uri=\"file:\/\/\/\/home\/customlexicon.xml\"\/>\r\n Please launch Teams.\r\n <\/voice>\r\n<\/speak><\/code><\/pre>\nExample SSML for Hybrid TTS (Cloud and Embedded)<\/h5>\n
<speak version=\"1.0\" xmlns=\"http:\/\/www.w3.org\/2001\/10\/synthesis\"\r\n xmlns:mstts=\"http:\/\/www.w3.org\/2001\/mstts\"\r\n xml:lang=\"en-US\">\r\n <voice name=\"en-US-JennyNeural\">\r\n <lexicon uri=\"https:\/\/example.com\/customlexicon.xml\"\/>\r\n <lexicon uri=\"file:\/\/\/\/home\/customlexicon.xml\"\/>\r\n Please launch Teams.\r\n <\/voice>\r\n<\/speak><\/code><\/pre>\n\n
file:\/\/\/\/<\/code> (four slashes) in SSML.<\/li>\n\n
Considerations and Best Practices<\/h3>\n
File Size Limits<\/h4>\n
\n
Caching Behavior<\/h4>\n
\n
Lexicon Validation<\/h4>\n
Recommended Phonetic Conversion Tools<\/h4>\n
\n
Custom Keyword Recognition<\/h2>\n
Self-Serve Portal Model<\/h3>\n
Implementation Steps<\/h4>\n
\n
.table<\/code> file.<\/li>\n