Wav2Letter RASR Model Test Results

There are so many toolkits and model releases that some interesting things left unnoticed. Some time ago Facebook published a small paper Rethinking Evaluation in ASR: Are Our Models Robust Enough? and released a set of robust US English models. They trained a big English conformer model on a number of datasets (librispeech, common voice, fisher, their private video dataset) total about 9k hours. And, I must say, to the date the model is one of the best ones.

I made a number of test on different datasets and here are the comparison of WER for existing Vosk models and RASR model.

Dataset Aspire Vosk model Daanzu Vosk model RASR model
Librispeech test-clean 11.72 7.08 3.30
Tedlium test 11.23 8.25 5.96
Google commands 46.76 11.64 20.06
Non-native speech 57.92 33.31 26.99
Children speech 20.29 9.90 6.17
Podcasts 19.85 21.21 15.06
Callcenter bot 17.20 19.22 14.55
Callcenter 1 53.98 52.97 42.82
Callcenter 2 33.82 43.02 30.41
Callcenter 3 35.86 52.80 32.98

I would say today this model is quite competitive. I also compared the model with few specialized models and while specialized models are better in terms of WER, RASR model probably can be finetuned to very good accuracy. I’m yet to try it.

On callcenter the model is not that shiny, very old and slow ASPIRE model is not significantly worse, but, again, given finetuning capabilities, it has a geat potential.

For videos/podcasts the accuracy is great, even for non-native speech.

The model requires new Flashlight library, not every easy to setup, I recommend to try it with a docker. Docker works smoothly.

The model is pretty big (2.5G acoustic model, 2.5G language model), but surprisingly it runs in a reasonable time even in CPU, so it an be used both in training and sometimes for decoding if good accuracy is required and speed is not an issue.

The paper doesn’t mention the training time for the models but I suspect it was quite long and required many GPUs, so such a model is not easily reproducible in a small organization. That makes the model even more valuable.

Thumbs up, Facebook!