Performance Disparities between Accents in Automatic Speech Recognition (Student Abstract)

Abstract

In this work, we expand the discussion of bias in Automatic Speech Recognition (ASR) through a large-scale audit. Using a large and global data set of speech, we perform an audit of some of the most popular English ASR services. We show that, even when controlling for multiple linguistic covariates, ASR service performance has a statistically significant relationship to the political alignment of the speaker's birth country with respect to the United States' geopolitical power.

Incorporating automatic speech recognition methods into the transcription of police-suspect interviews: factors affecting automatic performance

“…Factors relating to a speaker's linguistic background, such as accent, can prove challenging for an automatic transcription system. Previous work has demonstrated that the performance of ASR systems declines significantly when confronted with speech that diverges from the "standard" variety; this has been found for non-native-accented speech in English (Meyer et al, 2020;DiChristofano et al, 2022;Markl, 2022) and Dutch (Feng et al, 2021), as well as for non-standard regionallyaccented speech in Brazilian Portuguese (Lima et al, 2019) and British English (Markl, 2022). Markl (2022) compared the performance of Google and Amazon transcription services across multiple accents of British English.…”

“…In recent years, a growing body of research has focused on systematic bias within automatic systems, i.e., underperformance for certain demographic groups, and significant disparities in performance have been demonstrated across accents. Transcripts tend to be significantly less accurate for non-native speakers (DiChristofano et al, 2022) or speakers of nonstandard regional varieties (Markl, 2022). However, a limitation of work in this area is the use of word error rate (WER) for evaluating performance.…”