Revisiting Parity of Human vs. Machine Conversational Speech Transcription

Mansfield, Courtney; Ng, Sara; Levow, Gina‐Anne; Wright, Richard; Ostendorf, Mari

doi:10.21437/interspeech.2021-1908

Cited by 4 publications

(8 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Speech scientists have long worked to supplement word error rate with more informative measures, including error analyses of overlap (Çetin and Shriberg, 2006), disfluencies (Goldwater et al, 2010), and conversational words (Zayats et al, 2019;Mansfield et al, 2021). This work has shown the importance of in-depth error analysis, and also brings home the multi-faceted challenges of truly interactive speech-to-text systems.…”

Section: Related Workmentioning

confidence: 99%

“…The most widely used metric for comparison is word error rate, whose main attraction -simplicity-is also its most important pitfall. Here we build on prior work calling for error analysis beyond WER (Mansfield et al, 2021;Zayats et al, 2019) and extend it by looking at multiple languages and considering aspects of timing, confidence, conversational words, and dialog acts.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systems

Liesenfeld,

Lopez,

Dingemanse

2023

Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue

View full text Add to dashboard Cite

Speech recognition systems are a key intermediary in voice-driven human-computer interaction. Although speech recognition works well for pristine monologic audio, real-life use cases in open-ended interactive settings still present many challenges. We argue that timing is mission-critical for dialogue systems, and evaluate 5 major commercial ASR systems for their conversational and multilingual support. We find that word error rates for natural conversational data in 6 languages remain abysmal, and that overlap remains a key challenge (study 1). This impacts especially the recognition of conversational words (study 2), and in turn has dire consequences for downstream intent recognition (study 3). Our findings help to evaluate the current state of conversational ASR, contribute towards multidimensional error analysis and evaluation, and identify phenomena that need most attention on the way to build robust interactive speech technologies.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systems

Liesenfeld,

Lopez,

Dingemanse

2023

Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue

View full text Add to dashboard Cite

show abstract

“…A growing amount of work seeks to model feedback behaviour in human-agent interaction, including by means of response token generation [13,14] and attentive listening systems [15,16]. Despite considerable progress, the place of response tokens in speech technology is by no means settled: they tend to be missed by speech recognizers [17,18,19] and dialog managers have a hard time dealing with them [20], showing that they remain a key issue on which progress towards future generations of voice-interactive technologies and conversational user interfaces depends. Observational work on forms and functions of response tokens in human interaction is an important empirical foundation of any speech technology intended for human use.…”

Section: Related Workmentioning

confidence: 99%

Bottom-up discovery of structure and variation in response tokens (‘backchannels’) across diverse languages

Liesenfeld¹,

Dingemanse²

2022

Preprint

View full text Add to dashboard Cite

Response tokens (also known as backchannels, continuers, or feedback) are a frequent feature of human interaction, where they serve to display understanding and streamline turn-taking. We propose a bottom-up method to study responsive behaviour across 16 languages (8 language families). We use sequential context and recurrence of turns formats to identify candidate response tokens in a language-agnostic way across diverse conversational corpora. We then use UMAP clustering directly on speech signals to represent structure and variation. We find that (i) written orthographic annotations underrepresent the attested variation, (ii) distinctions between formats can be gradient rather than discrete, (iii) most languages appear to make available a broad distinction between a minimal nasal format ‘mm’ and a fuller ‘yeah’-like format. Charting this aspect of human interaction contributes to our understanding of interactional infrastructure across languages and can inform the design of speech technologies.

show abstract

“…Human transcribers disagree on this very difficult task [4,5,6,12,13], so it should not be surprising that there are inevitably some errors in these reference transcripts and mappings. For this work, a professional linguist was commissioned to very carefully audit and correct these references.…”

Section: Corrected Reference Filesmentioning

confidence: 99%

“…A careful analysis in [6] notes that "humans are more likely to miss words than to misrecognize them", and is notable in several regards: code was provided to specify a non-standard data cleaning and text normalization process, while output from a research system was re-scored in an (unsuccessful) attempt to replicate a published result. Our work continues in this effort to fully describe and improve upon the standard scoring methodology, sharing data and software to enable reproducible results.…”

Section: Introductionmentioning

confidence: 99%

Toward Zero Oracle Word Error Rate on the Switchboard Benchmark

Faria¹,

Janin²,

Riedhammer³

et al. 2022

Preprint

View full text Add to dashboard Cite

The "Switchboard benchmark" is a very well-known test set in automatic speech recognition (ASR) research, establishing record-setting performance for systems that claim human-level transcription accuracy. This work highlights lesser-known practical considerations of this evaluation, demonstrating major improvements in word error rate (WER) by correcting the reference transcriptions and deviating from the official scoring methodology. In this more detailed and reproducible scheme, even commercial ASR systems can score below 5% WER and the established record for a research system is lowered to 2.3%. An alternative metric of transcript precision is proposed, which does not penalize deletions and appears to be more discriminating for human vs. machine performance. While commercial ASR systems are still below this threshold, a research system is shown to clearly surpass the accuracy of commercial human speech recognition. This work also explores using standardized scoring tools to compute oracle WER by selecting the best among a list of alternatives. A phrase alternatives representation is compared to utterance-level N-best lists and word-level data structures; using dense lattices and adding out-of-vocabulary words, this achieves an oracle WER of 0.18%.

show abstract

Revisiting Parity of Human vs. Machine Conversational Speech Transcription

Cited by 4 publications

References 12 publications

The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systems

The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systems

Bottom-up discovery of structure and variation in response tokens (‘backchannels’) across diverse languages

Toward Zero Oracle Word Error Rate on the Switchboard Benchmark

Contact Info

Product

Resources

About