Retrospective Analysis of Clinical Performance of an Estonian Speech Recognition System for Radiology: Effects of Different Acoustic and Language Models

Abstract

The aim of this study was to analyze retrospectively the influence of different acoustic and language models in order to determine the most important effects to the clinical performance of an Estonian language-based non-commercial radiology-oriented automatic speech recognition (ASR) system. An ASR system was developed for Estonian language in radiology domain by utilizing open-source software components (Kaldi toolkit, Thrax). The ASR system was trained with the real radiology text reports and dictations collected during development phases. The final version of the ASR system was tested by 11 radiologists who dictated 219 reports in total, in spontaneous manner in a real clinical environment. The audio files collected in the final phase were used to measure the performance of different versions of the ASR system retrospectively. ASR system versions were evaluated by word error rate (WER) for each speaker and modality and by WER difference for the first and the last version of the ASR system. Total average WER for the final version throughout all material was improved from 18.4% of the first version (v1) to 5.8% of the last (v8) version which corresponds to relative improvement of 68.5%. WER improvement was strongly related to modality and radiologist. In summary, the performance of the final ASR system version was close to optimal, delivering similar results to all modalities and being independent on user, the complexity of the radiology reports, user experience, and speech characteristics.

Recognition of Creaky Voice from Emergency Calls

“…Although neural networks have been used for statistical modeling for decades, in recent years they have become increasingly utilized in the field of speech technology due to development of powerful GPUs. Using more computational power makes it possible to train DNN-based acoustic models on a very large scale, which has led to lower word error rate in comparison to GMMs even with small languages such as Estonian [16]. Yet, deep learning models are still rarely used for automatic voice quality detection in today's phonetic research.…”

RUTA:MED – Dual Workflow Medical Speech Transcription Pipeline and Editor

“…As for the ASR and post-processing part, we were inspired by the development of the Estonian ASR for Radiology [3], since Estonian is a similar-size language having comparable amount of language resources available, but again -RUTA:MED adds the dual workflow and interactive editing support.…”

Creation of Language Resources for the Development of a Medical Speech Recognition System for Latvian

“…Words that were not classified automatically need to be manually classified and transcribed. Words that occur at least 1,000 times in the text corpus (almost 14k words) 4 ,…”

“…ASR systems trained on general-purpose speech and text corpora, however, are not applicable 3 for the very specific language of medical reports. Domain-adapted language model and pronunciation lexicon (both derived from a text corpus of written medical reports) give the most significant boost in ASR accuracy, while a domain-adapted acoustic model (derived from a speech corpus of dictated medical reports) makes a considerable impact as well [4]. The work presented in this paper is a part of an ongoing collaborative project between a language technology research group and the largest hospital in Latvia on Latvian ASR for medical applications.…”

“…Moreover, transcriptionist services are expensive, and regional healthcare institutions cannot afford them. Inspired by the successful implementation of Estonian ASR for radiology [4], [5], our goal is to create the essential language resource and technology components for Latvian ASR in the medical domain, particularly in radiology, and showcase their usage by developing and validating an automated dictation platform for radiology reporting. The platform will be customised for two usage scenarios.…”