As everyone knows, ASR performance has improved dramatically in the last 10 years. The primary metric for ASR performance is “word error rate” (WER).\u00a0 Most folks think of WER as the percentage of words incorrectly recognized, although it’s not that simple.\u00a0 WER can be more than 1 (e.g., if you come up with a sentence given only noise!).\u00a0 Here is a comparison published in 2011.<\/p>\n
Today, Google, Amazon, Microsoft and others have WER under 10% in many cases. To get there, it takes some talent and thousands of hours of training data.\u00a0 Google is best, Alexa is close, and Microsoft lags a bit in 3rd place.\u00a0 (Click the graphic for the article summarizing Vocalize.io results.)<\/p>\n Even more impressive is that Google and Amazon are doing this with far-field microphones.\u00a0 Recognizing far-field speech is much harder than the clean speech that a near-field microphone, such as a high-quality headset microphone used in prior speech recognition benchmarking.<\/p>\n These numbers are better than Google and Amazon perform in regular usage tests, however.\u00a0 For example, using Alexa, you can say “Simon Says”, and it will read back what it thinks it heard.\u00a0 Try a couple 10 word sentences and you’ll probably conclude that Alexa has a WER of 10% or so (again, WER is not a percentage, but everyone treats it that way).\u00a0\u00a0 Google messes up roughly as much in my experience.\u00a0 I use Cortana (actually, Microsoft Cognitive Services, formerly known as Bing Speech) more and find it generally acceptable, although, as shown above, lagging behind Amazon and Google.\u00a0 I simply don’t know about Siri since I talk to array microphones of various sorts and at various distances, not into an iPhone mic or headset.<\/p>\n I suspect that the numbers above are unrealistically good from Amazon and Google because of the test cases.\u00a0 An utterance like “what time is it”, is pretty short and easy to model with a statistical language model.\u00a0\u00a0 It also helps that you prime the system by saying, “Alexa” or “Hey, whats-your-name”.\u00a0 More natural dialog without such “keyword spotting” is more challenging.\u00a0\u00a0\u00a0 You’re more like to say “uh” in a natural dialog than after you say, “Hey Google”.\u00a0 And, as you get into more natural utterances, they get longer and statistical language modelling does not help as much.<\/p>\n Indeed, statistical language modeling can increase errors.\u00a0 Language models are used in “decoding” the time-series of parameters estimated by an acoustic model which encodes the speech signal.\u00a0\u00a0\u00a0 Most ASRs decode using n-gram language models.\u00a0 Roughly speaking, if you have a language model that counts how many times each sequence of 4words is encountered in a corpus you can use it to guide or bias a best-first or beam-search towards a decoding that is “most consistent” with the corpus.\u00a0 If 4 words uttered do not occur in the corpus, decoding needs to estimate that 4-gram’s consistency with the corpus.\u00a0 The most common approach to doing so is by “backing off” to the consistency of 3 words and interpolating with respect to the 4th word.\u00a0 There’s been a lot of work in this area over several decades, but it’s been pretty stable since KenLM<\/a>, which is used in many state of are deep learning systems.\u00a0 The references of that paper lead to improved Knesser-Ney smoothing for the estimation problem.\u00a0 It is clear to me that Google does better.\u00a0 DeepSpeech uses KenLM and I observe the limitations of Knesser-Ney backoff leading to higher WER.\u00a0 As this has become quite technical, I will spare you the details!<\/p>\n It takes a lot of data to get anywhere near the far-field performance of Google and Amazon (or the near-field performance of Google and Apple), as shown by DeepSpeech numbers (again, click on the image for the PDF).<\/p>\n Google has a large edge over Amazon on data, thanks to its Android market penetration and tenure.\u00a0 Amazon had the lead with Echo, but Google has closed that gap this year.<\/p>\n All the rest of us working on ASR have data envy.\u00a0 In\u00a0 the table above, 80% of the data is Baidu’s!\u00a0 It’s hard for anyone to compete with these big players in WER metrics without many times the data that’s publicly available.\u00a0 Fortunately, we have the models made available by Baidu.\u00a0 Unfortunately, anyone who leverages their model is also stuck with it!<\/p>\n Fortunately, there a folks like Daniel Povey.\u00a0 He is behind KALDI (and an open spoken-language resources web site<\/a>).\u00a0 As you can see on “WER: where are we<\/a>“, his 2018 paper combining time-delay neural networks (TDNN) with KALDI<\/a> performs much better than DeepSpeech on the LibriSpeech test.\u00a0 This was accomplished with publicly available data (roughly the same first 20% shown above as used by Baidu).\u00a0 There are some caveats, however. As suggested above, its not just the acoustic model, but the language model impacts word error rate.\u00a0 This paper uses a language model that raises some questions…<\/p>\n Still, there is hope for more open ASR rivaling the leaders in terms of WER.\u00a0 The rest of the question is how well we’ll do at spoken language understanding<\/span>, which is enabled by low WER, just as natural language understanding (NLU) is enabled by better natural language processing<\/span> (NLP).\u00a0 It’s great fun to be closing the gap between ever-improving machine learning and more intelligent understanding.\u00a0\u00a0 Hopefully, this will lead to more valuable intelligent agents than the chatbots that are most likely to claim the Alexa Prize (where by chatbot I mean an agent whose primary goal is merely to converse).<\/p>\n","protected":false},"excerpt":{"rendered":" Some folks use the term “automatic speech recognition”, ASR.\u00a0 I don’t like the separation between recognition and understanding, but that’s where the technology stands. The term ASR encourages thinking about spoken language at a technical level in which purely inductive techniques are used to generate text from an audio signal (which is hopefully some recorded … <\/p>\n![\"\"](\"http:\/\/haleyai.com\/wordpress\/wp-content\/uploads\/SphinxBingGoogle.png\")
<\/a><\/p>\n![\"\"](\"http:\/\/haleyai.com\/wordpress\/wp-content\/uploads\/VocalizeSummary.png\")
<\/a><\/p>\n![\"\"](\"http:\/\/haleyai.com\/wordpress\/wp-content\/uploads\/DeepSpeech2TrainingData.png\")
<\/a><\/p>\n