ASR4REAL: An extended benchmark for speech models

Rivière, Morgane; Copet, Jade; Synnaeve, Gabriel

doi:10.48550/arxiv.2110.08583

Cited by 3 publications

(6 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, little investigation has been carried out about estimating the WERs gap produced by gender disparities. To name a few [15,16,17]. In this work, we perform an analysis by fine-tuning E2E models with ATC audio from different genders.…”

Section: Contribution and Motivationmentioning

confidence: 99%

“…All the experiments are based on the most robust E2E model from Table 2 i.e., w2v2-L-60K. 17 The WERs plot are obtained with greedy decoding and no LM or explicit textual information added. We fine-tune 18 models varying the training data set (either NATS or ISAVIA) and varying the amount of fine-tuning samples.…”

Section: Do Multilingual Pre-trained E2e Models Help?mentioning

confidence: 99%

“…Likewise, it is of special interest for the ATC community 17. We select the best model based on lowest WERs on out-of-domain test sets, i.e., ATCO2-Test and LiveATC-Test.…”

mentioning

confidence: 99%

See 2 more Smart Citations

How Does Pre-Trained Wav2Vec 2.0 Perform on Domain-Shifted Asr? an Extensive Benchmark on Air Traffic Control Communications

Zuluaga-Gómez

Prasad

Nigmatulina

et al. 2023

2022 IEEE Spoken Language Technology Workshop (SLT)

View full text Add to dashboard Cite

Recent work on self-supervised pre-training focus on leveraging large-scale unlabeled speech data to build robust end-to-end (E2E) acoustic models (AM) that can be later fine-tuned on downstream tasks e.g., automatic speech recognition (ASR). Yet, few works investigated the impact on performance when the data properties substantially differ between the pre-training and fine-tuning phases, termed domain shift. We target this scenario by analyzing the robustness of Wav2Vec 2.0 and XLS-R models on downstream ASR for a completely unseen domain, air traffic control (ATC) communications. We benchmark these two models on several open-source and challenging ATC databases with signal-to-noise ratio between 5 to 20 dB. Relative word error rate (WER) reductions between 20% to 40% are obtained in comparison to hybrid-based ASR baselines by only fine-tuning E2E acoustic models with a smaller fraction of labeled data. We analyze WERs on the low-resource scenario and gender bias carried by one ATC dataset.

show abstract

Section: Contribution and Motivationmentioning

confidence: 99%

Section: Do Multilingual Pre-trained E2e Models Help?mentioning

confidence: 99%

See 1 more Smart Citation

How Does Pre-Trained Wav2Vec 2.0 Perform on Domain-Shifted Asr? an Extensive Benchmark on Air Traffic Control Communications

Zuluaga-Gómez

Prasad

Nigmatulina

et al. 2023

2022 IEEE Spoken Language Technology Workshop (SLT)

View full text Add to dashboard Cite

show abstract

“…Previous research measuring bias in speech processing models largely studies differences in performance on specific speech tasks, for data sourced from people of differing social groups. Social group-based performance comparisons exist for Automated Speech Recognition (ASR) (Tatman, 2017;Tatman and Kasten, 2017;Koenecke et al, 2020;Feng et al, 2021;Liu et al, 2022b;Riviere et al, 2021), Speaker Verification or Speaker Identification (SID) (Hutiri and Ding, 2022;Fenu et al, 2021Fenu et al, , 2020Fenu and Marras, 2022;Chen et al, 2022b;Meng et al, 2022), as well as a number of other speech tasks (Meng et al, 2022;Hutiri et al, 2023). Differences in model performance based on the gender (Tatman, 2017;Tatman and Kasten, 2017;Chen et al, 2022b;Feng et al, 2021;Liu et al, 2022b;Hutiri and Ding, 2022;Fenu et al, 2020Fenu et al, , 2021Fenu and Marras, 2022;Riviere et al, 2021), dialect (Tatman, 2017;Tatman and Kasten, 2017), race (Koenecke et al, 2020;Tatman and Kasten, 2017;Chen et al, 2022b;Riviere et al, 2021), age (Fenu et al, 2020, city (Koenecke et al, 2020), nationality (Hutiri and Ding, 2022), and native language (Feng et al, 2021) of the speaker have been tested.…”

Section: Bias In Speech Modelsmentioning

confidence: 99%

Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognition

Slaughter,

Greenberg,

Schwartz

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Previous work has established that a person's demographics and speech style affect how well speech processing models perform for them. But where does this bias come from? In this work, we present the Speech Embedding Association Test (SpEAT), a method for detecting bias in one type of model used for many speech tasks: pre-trained models. The SpEAT is inspired by word embedding association tests in natural language processing, which quantify intrinsic bias in a model's representations of different concepts, such as race or valencesomething's pleasantness or unpleasantnessand capture the extent to which a model trained on large-scale socio-cultural data has learned human-like biases. Using the SpEAT, we test for six types of bias in 16 English speech models (including 4 models also trained on multilingual data), which come from the wav2vec 2.0, HuBERT, WavLM, and Whisper model families. We find that 14 or more models reveal positive valence (pleasantness) associations with abled people over disabled people, with European-Americans over African-Americans, with females over males, with U.S. accented speakers over non-U.S. accented speakers, and with younger people over older people. Beyond establishing that pre-trained speech models contain these biases, we also show that they can have real world effects. We compare biases found in pre-trained models to biases in downstream models adapted to the task of Speech Emotion Recognition (SER) and find that in 66 of the 96 tests performed (69%), the group that is more associated with positive valence as indicated by the SpEAT also tends to be predicted as speaking with higher valence by the downstream model. Our work provides evidence that, like text and image-based models, pretrained speech based-models frequently learn human-like biases when trained on large-scale socio-cultural datasets. Our work also shows that bias found in pre-trained models can propagate to the downstream task of SER.

show abstract

“…Research in fairness for speech recognition is still in its nascent stage, but is of great importance to society given the increasing pervasiveness of ASR technology. Prior studies have shown ASR performance disparities based on gender, age, race and ethnic backgrounds [8,9,10,11,12,13]. We hope to advance the work in this area with three contributions: (1) An assessment of fairness in speech recognition at scale; (2) a novel approach for automated discovery of underperforming cohorts; and (3) to the best of our knowledge, a first report on fairness mitigation for production-scale ASR systems.…”

Section: Introductionmentioning

confidence: 95%

Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparities

Dheram¹,

Ramakrishnan²,

Raju³

et al. 2022

Preprint

View full text Add to dashboard Cite

As for other forms of AI, speech recognition has recently been examined with respect to performance disparities across different user cohorts. One approach to achieve fairness in speech recognition is to (1) identify speaker cohorts that suffer from subpar performance and (2) apply fairness mitigation measures targeting the cohorts discovered. In this paper, we report on initial findings with both discovery and mitigation of performance disparities using data from a product-scale AI assistant speech recognition system. We compare cohort discovery based on geographic and demographic information to a more scalable method that groups speakers without human labels, using speaker embedding technology. For fairness mitigation, we find that oversampling of underrepresented cohorts, as well as modeling speaker cohort membership by additional input variables, reduces the gap between top-and bottom-performing cohorts, without deteriorating overall recognition accuracy.

show abstract

ASR4REAL: An extended benchmark for speech models

Cited by 3 publications

References 14 publications

How Does Pre-Trained Wav2Vec 2.0 Perform on Domain-Shifted Asr? an Extensive Benchmark on Air Traffic Control Communications

How Does Pre-Trained Wav2Vec 2.0 Perform on Domain-Shifted Asr? an Extensive Benchmark on Air Traffic Control Communications

Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognition

Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparities

Contact Info

Product

Resources

About