Don't Speak Too Fast: The Impact of Data Bias on Self-Supervised Speech Models

Meng, Yen; Chou, Yi-Hui; Liu, Andy T.; Lee, Hung-yi

doi:10.1109/icassp43922.2022.9747897

Cited by 5 publications

(8 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other corpora exist with transcript ground-truth, typically to train voicerecognition systems, e.g., Common Voice (Ardila et al 2020) and Artie Bias (Meyer et al 2020), but they are not suitable to study transcription bias due to the short sentences and the likelihood that they have been already used to train those systems. (2020); Buolamwini and Gebru (2018)), where face recognition bias is evaluated for several personal characteristics; smart speakers wake-word recognition (e.g., Chen et al (2021); Dubois et al (2020)), where the authors show that certain categories of people are more likely to misactivate popular smart speakers; automated speaker recognition (Hutiri and Ding 2022;Fenu et al 2021;Meng et al 2022;Hajavi and Etemad 2023), where the authors analyze the fairness in recognizing speakers.…”

Section: Related Workmentioning

confidence: 99%

Fair or Fare? Understanding Automated Transcription Error Bias in Social Media and Videoconferencing Platforms

Dubois,

Holliday,

Waddell

et al. 2024

ICWSM

View full text Add to dashboard Cite

As remote work and learning increases in popularity, individuals, especially those with hearing impairments or who speak English as a second language, may depend on automated transcriptions to participate in business, school, entertainment, or basic communication. In this work, we investigate the automated transcription accuracy of seven popular social media and videoconferencing platforms with respect to some personal characteristics of their users, including gender, age, race, first language, speech rate, F0 frequency, and speech readability. We performed this investigation on a new corpus of 194 hours of English monologues by 846 TED talk speakers. Our results show the presence of significant bias, with transcripts less accurate for speakers that are male or non-native English speakers. We also observe differences in accuracy among platforms for different types of speakers. These results indicate that, while platforms have improved their automatic captioning, much work remains to make captions accessible for a wider variety of speakers and listeners.

show abstract

Section: Related Workmentioning

confidence: 99%

Fair or Fare? Understanding Automated Transcription Error Bias in Social Media and Videoconferencing Platforms

Dubois,

Holliday,

Waddell

et al. 2024

ICWSM

View full text Add to dashboard Cite

show abstract

“…However, little investigation has been carried out about estimating the WERs gap produced by gender disparities. To name a few [15,16,17]. In this work, we perform an analysis by fine-tuning E2E models with ATC audio from different genders.…”

Section: Contribution and Motivationmentioning

confidence: 99%

“…To address this question, we only focus on w2v2-L-60k and w2v2-L-60k+ models fine-tuned on the 32 hrs and 132 hrs sets, respectively. 15 . We analyze the WERs obtained by greedy decoding to focus only on joint acoustic and language ASR modeling (see Section 2).…”

Section: Unlabeledmentioning

confidence: 99%

“…Gender bias on air traffic control communications: finally, it is becoming a trend to go beyond simply analyzing the performance of E2E models on the selected downstream task, e.g., ASR. For instance, examining the performance shift of a given ASR model when fine-tuned on audio from different genders [15,16]. We analyze this bias on ATCOSIM dataset, which provide the gender labels for each utterance.…”

Section: Do Multilingual Pre-trained E2e Models Help?mentioning

confidence: 99%

“…For instance, previous work have concluded that gender unbalance might affect E2E models during the pre-training phase. However, this bias can be mitigated by adding a small amount of data from the opposite gender [15]. In conclusion, it is still prudent to perform more thorough experiments before reaching hard judgments in this regard, or at least, in ATC communications.…”

Section: Do Multilingual Pre-trained E2e Models Help?mentioning

confidence: 99%

See 2 more Smart Citations

How Does Pre-Trained Wav2Vec 2.0 Perform on Domain-Shifted Asr? an Extensive Benchmark on Air Traffic Control Communications

Zuluaga-Gómez

Prasad

Nigmatulina

et al. 2023

2022 IEEE Spoken Language Technology Workshop (SLT)

View full text Add to dashboard Cite

Recent work on self-supervised pre-training focus on leveraging large-scale unlabeled speech data to build robust end-to-end (E2E) acoustic models (AM) that can be later fine-tuned on downstream tasks e.g., automatic speech recognition (ASR). Yet, few works investigated the impact on performance when the data properties substantially differ between the pre-training and fine-tuning phases, termed domain shift. We target this scenario by analyzing the robustness of Wav2Vec 2.0 and XLS-R models on downstream ASR for a completely unseen domain, air traffic control (ATC) communications. We benchmark these two models on several open-source and challenging ATC databases with signal-to-noise ratio between 5 to 20 dB. Relative word error rate (WER) reductions between 20% to 40% are obtained in comparison to hybrid-based ASR baselines by only fine-tuning E2E acoustic models with a smaller fraction of labeled data. We analyze WERs on the low-resource scenario and gender bias carried by one ATC dataset.

show abstract

What makes things catch on? Understanding consumer engagement with video content on social media

Kim

2024

Electron Commer Res

View full text Add to dashboard Cite

Don't Speak Too Fast: The Impact of Data Bias on Self-Supervised Speech Models

Cited by 5 publications

References 23 publications

Fair or Fare? Understanding Automated Transcription Error Bias in Social Media and Videoconferencing Platforms

Fair or Fare? Understanding Automated Transcription Error Bias in Social Media and Videoconferencing Platforms

How Does Pre-Trained Wav2Vec 2.0 Perform on Domain-Shifted Asr? an Extensive Benchmark on Air Traffic Control Communications

What makes things catch on? Understanding consumer engagement with video content on social media

Contact Info

Product

Resources

About