Automatic human utility evaluation of ASR systems: does WER really predict performance?

Favre, Benoît; Cheung, K.M.; Kazemian, Siavash; Lee, Adam F.; Liu, Yang; Munteanu, Cosmin; Nenkova, Ani; D, Ochei; Penn, Gerald; Tratz, Stephen; Voss, Clare R.; Zeller, Frauke

doi:10.21437/interspeech.2013-610

Cited by 22 publications

(11 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Le WER se calculera de la manière suivante : Bien qu'il soit le plus utilisé, le WER présente de nombreuses lacunes (Favre et al, 2013 ;Morris et al, 2004 ;Nanjo et al, 2005 ;Park et al, 2008 ;Wang et al, 2003). Tout d'abord, le WER est un taux qui n'a pas de limite supérieure : le nombre d'ajouts étant illimité, le pourcentage d'erreurs de mots peut être supérieur à 100 %, ce qui ne permet pas de savoir si le système est bon, mais seulement s'il est meilleur qu'un autre.…”

Section: Mesure La Plus Utilisée : Le Werunclassified

See 1 more Smart Citation

Un mot pour un autre ? Analyse et comparaison de huit plateformes de transcription automatique

Tancoigne

Corbellini

Deletraz

et al. 2022

Bulletin of Sociological Methodology/Bulletin de Méthodologie S

View full text Add to dashboard Cite

One word for another? Analysis and comparison of eight automatic transcription platforms. This article compares the functionalities and results of eight automatic transcription platforms (Go Transcribe, Happy Scribe, Headliner, Sonix, Video Indexer, Vocalmatic, Vocapia and YouTube), for audio samples in French. We propose an original methodology, designed through an interdisciplinary work, to compare the transcriptions. It combines three complementary approaches: (1) a quantitative approach which compares the textual outcomes using a common metric, the Word Error Rate (WER), (2) a fine-grained approach to classify and understand the errors generated by the platforms, and finally (3) an approach estimating the amount of transcription time which can be saved for each file on each platform. We show that no platform surpassed the others for all the samples, but two nevertheless stood out: Vocapia and Sonix, each with their own areas of expertise. Regardless of the type of file or platform, listening and correcting the text remains a necessary step. However the use of such tools can save up to 75% of time compared with manual transcription. Yet, the use of these online tools can create major problems relating to data confidentiality and security. Finally, we reflect on the interdisciplinary setting that made this project possible.

show abstract

Section: Mesure La Plus Utilisée : Le Werunclassified

“…Bien qu'il soit le plus utilisé, le WER présente de nombreuses lacunes (Favre et al, 2013 ;Morris et al, 2004 ;Nanjo et al, 2005 ;Park et al, 2008 ;Wang et al, 2003). Tout…”

Section: Mesure La Plus Utilisée : Le Werunclassified

Un mot pour un autre ? Analyse et comparaison de huit plateformes de transcription automatique

Tancoigne

Corbellini

Deletraz

et al. 2022

Bulletin of Sociological Methodology/Bulletin de Méthodologie S

View full text Add to dashboard Cite

show abstract

“…However, none seem to have been used to the same extent as WER despite criticism from many researchers such as those mentioned by Kuhn, Kersken and Zimmermann who add that "ASR output should be validated for real-world use-cases" [5]. So, although WER may provide insights into the accuracy levels of automatic captions and transcriptions [6] there remain other factors that can be explored to provide additional feedback for ASR training models.…”

Section: Introductionmentioning

confidence: 99%

Exploring Practical Metrics to Support Automatic Speech Recognition Evaluations

Draffan,

Wald,

Ding

et al. 2023

Assistive Technology: Shaping a Sustainable and Inclusive World

View full text Add to dashboard Cite

Recent studies into the evaluation of automatic speech recognition for its quality of output in the form of text have shown that using word error rate to see how many mistakes exist in English does not necessarily help the developer of automatic transcriptions or captions. Confidence levels as to the type of errors being made remain low because mistranslations from speech to text are not always captured with a note that details the reason for the error. There have been situations in higher education where students requiring captions and transcriptions have found that some academic lecture results are littered with word errors which means that comprehension levels drop and those with cognitive, physical and sensory disabilities are particularly affected. Despite the incredible improvements in general understanding of conversational automatic speech recognition, academic situations tend to include numerous domain specific terms and the lecturers may be non-native speakers, coping with recording technology in noisy situations. This paper aims to discuss the way additional metrics are used to capture issues and feedback into the machine learning process to enable enhanced quality of output and more inclusive practices for those using virtual conferencing systems. The process goes beyond what is expressed and examines paralinguistic aspects such as timing, intonation, voice quality and speech understanding.

show abstract

“…WER is calculated from a literal word-level comparison of ground-truth with the hypothesis. There is an increasing interest among the speech community to consider an alternate evaluation metric [1,2], since WER does not capture the semantic correctness of the hypothesis generated. Consider the ground-truth as 'I am going to die', and the two hypotheses from ASR be 'I m going to die' and 'I am going to live'.…”

Section: Introductionmentioning

confidence: 99%

Hybrid-SD (H_SD): A new hybrid evaluation metric for automatic speech recognition tasks

Sasindran¹,

Yelchuri²,

Rao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Many studies have examined the shortcomings of word error rate (WER) as an evaluation metric for automatic speech recognition (ASR) systems, particularly when used for spoken language understanding tasks such as intent recognition and dialogue systems. In this paper, we propose Hybrid-SD (H SD ), a new hybrid evaluation metric for ASR systems that takes into account both semantic correctness and error rate. To generate sentence dissimilarity scores (SD), we built a fast and lightweight SNanoBERT model using distillation techniques. Our experiments show that the SNanoBERT model is 25.9x smaller and 38.8x faster than SRoBERTa while achieving comparable results on well-known benchmarks. Hence, making it suitable for deploying with ASR models on edge devices. We also show that H SD correlates more strongly with downstream tasks such as intent recognition and named-entity recognition (NER).

show abstract

Automatic human utility evaluation of ASR systems: does WER really predict performance?

Cited by 22 publications

References 13 publications

Un mot pour un autre ? Analyse et comparaison de huit plateformes de transcription automatique

Un mot pour un autre ? Analyse et comparaison de huit plateformes de transcription automatique

Exploring Practical Metrics to Support Automatic Speech Recognition Evaluations

Hybrid-SD (H_SD): A new hybrid evaluation metric for automatic speech recognition tasks

Contact Info

Product

Resources

About