How to evaluate ASR output for named entity recognition?

Jannet, Mohamed Ameur Ben; Galibert, Olivier; Adda-Decker, Martine; Rosset, Sophie

doi:10.21437/interspeech.2015-322

Cited by 11 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To conclude, this study presents promising results in a first attempt to experiment with the E2E approach to recognize named entities and constitutes an interesting start point for future work. In the future, we can study the effect of other loss metrics including: NE-WER (Named Entity Word Error Rate) [30] and ATENE (Automatic Transcription Evaluation for Named Entity) [31], instead of just using WER. Additionally, we can further work on our discussions on handling OOV words in an ASR system.…”

Section: Discussionmentioning

confidence: 99%

End-to-End Named Entity Recognition from English Speech

Yadav

Ghosh

et al. 2020

Interspeech 2020

View full text Add to dashboard Cite

Named entity recognition (NER) from text has been a widely studied problem and usually extracts semantic information from text. Until now, NER from speech is mostly studied in a twostep pipeline process that includes first applying an automatic speech recognition (ASR) system on an audio sample and then passing the predicted transcript to a NER tagger. In such cases, the error does not propagate from one step to another as both the tasks are not optimized in an end-to-end (E2E) fashion. Recent studies confirm that integrated approaches (e.g., E2E ASR) outperform sequential ones (e.g., phoneme based ASR). In this paper, we introduce a first publicly available NER annotated dataset for English speech and present an E2E approach, which jointly optimizes the ASR and NER tagger components. Experimental results show that the proposed E2E approach outperforms the classical two-step approach. We also discuss how NER from speech can be used to handle out of vocabulary (OOV) words in an ASR system.

show abstract

Section: Discussionmentioning

confidence: 99%

End-to-End Named Entity Recognition from English Speech

Yadav

Ghosh

et al. 2020

Interspeech 2020

View full text Add to dashboard Cite

show abstract

“…Our experiments were carried out on the French QUAERO data, officially used in the framework of the ETAPE evaluation campaign [13] and in different research studies [14,15] dedicated to named entity recognition from speech.…”

Section: Methodsmentioning

confidence: 99%

End-to-end model for named entity recognition from speech without paired training data

Mdhaffar¹,

Duret²,

Parcollet³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent works showed that end-to-end neural approaches tend to become very popular for spoken language understanding (SLU). Through the term end-to-end, one considers the use of a single model optimized to extract semantic information directly from the speech signal. A major issue for such models is the lack of paired audio and textual data with semantic annotation. In this paper, we propose an approach to build an end-to-end neural model to extract semantic information in a scenario in which zero paired audio data is available. Our approach is based on the use of an external model trained to generate a sequence of vectorial representations from text. These representations mimic the hidden representations that could be generated inside an end-to-end automatic speech recognition (ASR) model by processing a speech signal. A SLU neural module is then trained using these representations as input and the annotated text as output. Last, the SLU module replaces the top layers of the ASR model to achieve the construction of the end-to-end model. Our experiments on named entity recognition, carried out on the QUAERO corpus, show that this approach is very promising, getting better results than a comparable cascade approach or than the use of synthetic voices.

show abstract

“…The limitation of SWER is the need for an annotated reference. In addition, domain-specific metrics [3,4,5] have been proposed as a means of evaluating ASR systems such that it better reflects the system's performance on various NLU/NLP downstream tasks. However, the disadvantage of these methods is their inability to generalize for various applications.…”

Section: Introductionmentioning

confidence: 99%

Perceptual and Task-Oriented Assessment of a Semantic Metric for ASR Evaluation

Rugayan¹,

Salvi²,

Svendsen³

2023

Interspeech 2023

View full text Add to dashboard Cite

Automatic speech recognition (ASR) systems have become a vital part of our everyday lives through their many applications. However, as much as we have developed in this regard, our most common evaluation method for ASR systems still remains to be word error rate (WER). WER does not give information on the severity of errors, which strongly impacts practical performance. As such, we examine a semantic-based metric called Aligned Semantic Distance (ASD) against WER and demonstrate its advantage over WER in two facets. First, we conduct a survey asking participants to score reference text and ASR transcription pairs. We perform a correlation analysis and show that ASD is more correlated to the human evaluation scores compared to WER. We also explore the feasibility of predicting human perception using ASD. Second, we demonstrate that ASD is more effective than WER as an indicator of performance on downstream NLP tasks such as named entity recognition and sentiment classification.

show abstract

How to evaluate ASR output for named entity recognition?

Cited by 11 publications

References 0 publications

End-to-End Named Entity Recognition from English Speech

End-to-End Named Entity Recognition from English Speech

End-to-end model for named entity recognition from speech without paired training data

Perceptual and Task-Oriented Assessment of a Semantic Metric for ASR Evaluation

Contact Info

Product

Resources

About