OpenAI Whisper Accuracy and other recent models (Nemo Transducer XLarge, Gigaspeech)

Everyone is crazy about OpenAI Whisper. Trained on 680 thousand hours of multilingual data it indeed sets a new stage in speech recognition.

We tested it and some other recent models a bit on the same data we tested Nemo Conformer and other models before. Here are the results.

Some afterthoughs:

• Models are impressive
• More surprising is that en tiny models are pretty good, degradation from pruning is not that significant
• Other models are not that bad too like Gigaspeech models from Wenet and K2 or Nemo models, if you look for best accuracy they are about the same
• Wenet gigaspeech model (with LM rescoring) is not that bad
• RTX varies with K2 being the fastest
• Whisper LID model is not very good, on non-native speech it often confuses languages
• Whisper postprocessing is nice, but often arbitrary in decisions as well
• Whisper has funny hallucinations on short commands
• Not all properties of whisper models are fully understood yet, first robustness to noise (we heard echo harms a lot)
• Yet to test improvement on longer files compared to utterance-based processing of other systems (this is one of the strongest point in Whisper)
• Yet to test accuracy on other important languages systematically
• Vosk accuracy is outdated, need to improve ASAP