Voting, Ensembles and bringing AI to life
While people recently argue if Google’s model is sentient we must admit
that another important property of the living creatures emerged in recent AI models - they started to have experience. They see new data, new tests,
and improve over time. They teach each other like K2 model learns from Hubert. Vosk learns from NVIDIA Nemo. And this
experience is not really explainable. You can not really understand what is going on inside the latest Nemo conformer model
but you still can learn from the predictions it is making.
There are important consequences of treating AI models as live creatures with experience, for example, it changes my view on using system combinations.
Long ago (in 2013) I was thinking that system combination is not a
reasonable idea and it doesn’t bring anything
I was always blaming Kaggle competitions for ensembling models. I thought
that single best model is possible. I used to think that it is not such a
great idea to have 10 projects dedicated to speech recognition but recent
developments proved me wrong (again).
The core math here is actually very simple. If you have a complex
multidimensional space and you want to explore it, you can not make any
reasonable loop over points of the space. It is either too complex or too
slow. The only way is to drop some random points and hope that their
estimate will show something. It is not guaranteed that you get the
results but often you can get very reasonable estimate quickly. It is an
idea of Monte-Carlo computation and also an idea of many other randomized
algorithms which are abundant. Not always the best solution but a
solution with many nice properties.
First, you get an estimate quickly. Second, you can actually get not just
the first order estimate but also second order estimate (deviation or
confidence) which is very important for applications. And in many cases
like speech recognition it is the only way to get proper confidence (see Uncertainty Estimation in Autoregressive Structured Prediction).
Yeah, that diversity and democracy enforced in Western universities,
sometimes it makes sense. Of course sometimes it is not the best solution,
a single model is right and majority wrong but in many cases it really
makes sense to vote.
If we treat speech recognition models this way, few items emerge:
ROVER algorithm and
many other algorithms for system combination get more and more
Instead of HF style we don’t simply put models on one page but we
need to build tools to learn from each model, perform cross-confirmation and
Continuous evaluation of tools and models is also becoming relevant. Here
it is worth to mention SpeechIO leaderboard, or recent paper on evaluation of different cloud
providers on Switchboard
There will be no single best toolkit anymore, some models will be good for one domain, others for another. Unless we build a meta-toolkit (remember Turchin’s metasystems).
We consider this direction important for Vosk and industry in general.