Testing CMUSphinx with Hudson

As every high-quality product CMUSphinx spend a lot on testing. That isn't really trivial task because you need to make sure that all parameters that are important are improved or at least not regressed. That includes decoding accuracy, speed and API specs. Sometimes changes improve one thing and make other worse. Things are going to change with the deployment of continuous integration system Hudson.

Quite sophisticated system of tests was created to track changes. That included perl scripts, various shell bits, mysql database and even commits to CVS repository. It was also spamming mailing list all the day with long and unreadable emails. Another bad thing was that it's based on private commercial data like WSJ or TIDIGITS database but now everything is changing with Voxforge test set. Our goal is to let you test and optimize system yourself


.

Luckily, it went down recently. I was wondering about why I don't get any mails for almost a month but I'm pretty sure whole thing just not working anymore. There is a good side then since we have more reasons to move to something that specifically developed to support testing workflow. With Peter's hint I downloaded and instaled Hudson.

It's a nice thing in open source that you can get really professional product designed specifically to solve your problems by experts who understand what your problems are. I very much disagree with the people when they think they can write something themselves from scratch. I never understood this point of view.

Anyway, Hudson is our testing guard right now, you shouldn't worry about regressions anymore. Its running all CMUSphinx tests and some accuracy tests as well. It also builds nice graphs (see on the right) and can present you statistics about last builds.



 This piece of software is very good. What do I like about it:

• Slick interface
• Many plugins (I hate plugins to be honest, but hudson's plugins are great)
• Customizable build process without any restrictions  - you can run shell scripts, you can run your own apps, basically you can do everything
• Intelligent notification. Thanks Hudson, you only send me email when something is broken, no need to read mails every day!

It's defiitely a step forward in testing.
This server is private right now but we hope to make it public soon. Oh, and you can probably see there was a regression on July 1st that I committed for sphinx4. Accuracy dropped significantly and speed become a little bit faster. If you are tracking cmusphinx-devel, you probably know what is it about. Let's hope we'll fix it soon and than it will be really great thing to have.