Confidence Intervals for ASR-Based TTS Evaluation

Abstract

Automatic speech recognition (ASR) is increasingly used to evaluate the intelligibility of text-to-speech synthesis (TTS). ASR is less costly than traditional listening tests, but questions remain about its reliability. We re-evaluate the Blizzard Challenge's intelligibility tasks in English since 2011 using ASR. Re-analysing transcriptions collected by paid in-lab participants, online volunteers and Amazon Mechanical Turkers (the latter used only in 2011), we compare their word error rates (WERs) and statistically-significant system-groupings with those generated by an open-source, Transformer-based ASR model. This ASR model consistently decodes test stimuli with more reliable WERs than the Blizzard Challenge's (mostly non-native) speech experts and online volunteers. The model also groups systems according to statistical significance similarly to the paid in-lab participants. Using surplus semantically unpredictable sentences (SUS) submitted every year to the challenge, we investigate how confidence intervals in ASR WERs change as the number of transcribed stimuli increases. We plot the Frobenius norm of pairwise significance matrices with increasing stimuli. We find that finer groupings of systems are detected as confidence intervals narrow. The number of stimuli where p-values start to converge ranges from 400-800 stimuli. We conclude that, with enough stimuli, ASR can be more reliable than humans.

Building Synthetic Speaker Profiles in Text-to-Speech Systems

“…To evaluate the intelligibility of synthesized speech signals, we use the word error rate (WER) from Amazon speech recognition system 2 . As shown in [19], ASR-based metrics, e.g. WER, can perform reliably for evaluating intelligibility of TTS systems, and indeed on a comparable level to the paid human annotators/listeners.…”

“…To prevent this, we use the intelligibility metric to measure the quality of synthetic speaker profiles and synthesized speech signals. Similar to [19], we use the word error rate (WER) of a given speech recognition model as the intelligibility: when the WER is smaller, it means the intelligibility of synthesized speech signals is better.…”

FlexLip: A Controllable Text-to-Lip System

“…To evaluate the TTS systems, we analysed the objective measures of word error rate (WER) and a speaker-encoder-based cosine similarity over the entire Obama test set. These two measures have recently been found to have a high correlation to the perceptual measures obtained in listening tests [29,30]. The WERs are extracted based on the automatic transcripts provided by the SpeechBrain ASR system [22], while the cosine similarity uses SpeechBrain's speaker embedding network.…”