Digging the acoustic model bug

It's so useful to sleep a lot and care about the details. Today I've found the bug in the model I think I could search for years. Accidentally the topology of the model was wrong and I was using 5-state hmm without the transition from state number n to state n+2. The result was quite bad recognition accuracy. This issue was so well hidden I wonder if it's possible to discover such things at all.

The method that helped is related to comparative debugging. By checking the performance of pocketsphinx and sphinx4 I found they differ. But it wasn't enough. The critical point that helped is the unrelated change in pocketsphinx. Previously it assigned very small non-zero transition probabilities even if model had zero transitions. Happily, David changed that about a month ago. And the difference in recognition rate between recent and older version of pocketsphinx helped to find the problem. I was really lucky today.

I wonder how diagnostics could be extended here to help to resolve issues like that. It seems for me extremely important to build a recognition system that allows verification. The similar problem came today on another front btw. Our router had issue in DNS relay, but it was almost impossible to discover the reason due to limited diagnostic output. We really need to rethink the current way of error reporting.