In-depth evaluation of ASR engines

While everyone focuses on latency, there are many measurable aspects of ASR that are easy to evaluate and have a significant impact on user experience. Here are some of them:

1. Hallucination rates from noisy inputs. This is really useful for estimation how robust your recognizer is. We just feed 1000 noisy samples into every engine and measure how many words it produces. Some systems hallucinate significantly, exceeding 100% of insertions, some just give a few extra words. Very easy to measure, very useful in practice.

2. Recognition of the short inputs. While everyone trains on longer samples, short inputs sometimes gets less attention as a result systems have very big trouble to recognize simple “yes”. A huge thing for voicebots, much more important than latency. We specifically feed real-life short data and compare engines, results might be very different from system to system.

3. Ability to identify non-speech sounds as music and noises. While most modern systems become more and more intelligient the users expect them to react to a wide range of conditions. Even embedded systems need to identify the environment around properly. Very few engines able to do that.

4. Rare words problem. Because end-to-end systems learn distributions as is, the hidden distributions mismatches becomes a hidden but serious problems. For example, end-to-end systems learn to recognize frequent words well but our semantics obviously depends a lot on rare words - names, street names, products, medications. End-to-end systems have trouble with them, even the one trained with millions hours of data. People use LLM to estimate WER, talk about BERT WER but the reality is simple. You drop 30k most frequent words and consider substituion rate for the remaining tail. Take the standard Tedlium test and two modern English systems - Cohere and Qwen3-ASR.

You see even top systems with different architecture (encoder-decoder and LLM) make mistakes on rare words much more often than on common words. A paper like Improving accuracy of rare words for RNN-Transducer through unigram shallow fusion might be a great start as well as subsequent publications.

Recently we worked a lot on Russian ASR and we can introduce new version Vosk 0.62 focused exactly on the problems above. You might see the accuracy) didn’t improve much but performance in noise, rare words and music detection imroved significantly.

Extra features

Despite not being top accuracy we still see our models improve user experience significantly.

Latency and WER are, of course, still important, and we need to incorporate latency more systematically into our standard tests. Hopefully, we will also develop additional approaches for handling rare words, although the problem require much deeper architecture changes and brings us back to the LM integration issues discussed in the previous post. More on it later.