Testing Facebook MMS and SeamlessMT4 Word Error Rate
Recently Facebook released MMS and Seamless models for multilingual ASR and translation
Releases got some coverage in media and frequently mentioned as a cool model on Github.
Claims from the paper they are
very accurate, more accurate than Whisper V2 on Fleurs dataset.
Testing on standard datasets shows that models are a bit worse than other
public models. Moreover, if we talk about Seamless model specifically, it
not just have worse WER, it also replaces the words with synonyms, for
reference : experience pre-meal kit
hypothesis : experience before meal
It is ok for translation but for ASR task it doesn’t seem right. Seamless model is not very
practical outside translation because of that.
Still, accuracy is not bad and the models are impressive given they are
multilingual and can do translation well.
||MMS 1B + LM
||Sherpa Multi EN 05.2023
|Librispeech Test Clean