Researchers’ information set pinpoints challenges adapting speech recognition fashions to new {hardware}

Researchers’ information set pinpoints challenges adapting speech recognition fashions to new {hardware}


A new research from researchers affiliated with the College School London, Nokia Bell Labs Cambridge, and the College of Oxford reveals how variations in microphone high quality can impression speech recognition accuracy. In it, the coauthors use a customized information set — Libri-Adapt — containing 7,200 hours of English speech to check whether or not Mozilla’s DeepSpeech mannequin handles distinctive environments and microphones effectively. The findings recommend there’s a noticeable degradation in accuracy throughout sure “area shifts,” with phrase error charge rising to as excessive as 28% after switching microphones.

Computerized speech recognition fashions should carry out effectively throughout {hardware} in the event that they’re to be dependable. As an illustration, prospects count on the fashions powering Alexa to work equally on totally different good audio system, good shows, and good gadgets. However not all fashions obtain this splendid as a result of they’re not persistently skilled with various corpora. That’s to say, some corpora don’t comprise speech recorded with microphones of various high quality and in novel settings.

Libri-Adapt is designed to reveal these flaws with speech recorded utilizing the microphones in six totally different merchandise: A PlayStation Eye digicam, a generic USB mic, a Google Nexus 6 smartphone, the Shure MV5, a Raspberry Pi accent referred to as ReSpeaker, and the Matrix Voice developer equipment. The corpus has speech information in three English accents, specifically U.S. English, British English, and Indian English, which got here from 251 U.S. audio system and artificial voices generated by Google Cloud Platform’s text-to-speech API. Past this, Libra-Adapt comprises wind, rain, and laughter background noises meant to function added confounders.

Above: Phrase error charge of a fine-tuned DeepSpeech mannequin skilled and examined on varied microphone pairs for U.S. English speech. The columns correspond to the coaching microphone area and rows correspond to the take a look at microphone area.

Throughout experiments, the researchers in contrast the speech recognition efficiency of a pre-trained DeepSpeech mannequin (model 0.5.0) throughout the aforementioned six gadgets. They discovered that when information from the identical microphone was used for coaching and testing the mannequin, DeepSpeech unsurprisingly achieved the smallest error charge (e.g., 11.39% within the case of PlayStation Eye). However the inverse was additionally true: When there was a mismatch between the coaching and testing units, the phrase error charge jumped considerably (e.g., 24.18% when a mannequin skilled on PlayStation Eye-recorded speech was examined on Matrix Voice speech).

The researchers say that Libra-Adapt, which is offered in open supply, can be utilized to create eventualities that take a look at the generalizability of speech recognition algorithms. For example, they examined a DeepSpeech mannequin skilled on U.S.-accented speech collected by a ReSpeaker microphone in opposition to Indian-accented speech with rain background noise recorded by a PlayStation Eye. The outcomes present the mannequin suffered an error charge uptick of practically 29.8%, pointing to poor robustness on the mannequin’s half.

Though the coauthors declare to have manually verified a whole bunch of Libra-Adapt’s recordings, they warning that some is perhaps incomplete or noisy. That’s the rationale why they plan to develop unsupervised area adaptation algorithms in future work to sort out area shifts within the information set.

Leave a Reply