Robust Prediction of Punctuation and Truecasing for Medical ASR

Abstract

Automatic speech recognition (ASR) systems in the medical domain that focus on transcribing clinical dictations and doctor-patient conversations often pose many challenges due to the complexity of the domain. ASR output typically undergoes automatic punctuation to enable users to speak naturally, without having to vocalise awkward and explicit punctuation commands, such as "period", "add comma" or "exclamation point", while truecasing enhances user readability and improves the performance of downstream NLP tasks. This paper proposes a conditional joint modeling framework for prediction of punctuation and truecasing using pretrained masked language models such as BERT, BioBERT and RoBERTa. We also present techniques for domain and task specific adaptation by finetuning masked language models with medical domain data. Finally, we improve the robustness of the model against common errors made in ASR by performing data augmentation. Experiments performed on dictation and conversational style corpora show that our proposed model achieves ∼5% absolute improvement on ground truth text and ∼10% improvement on ASR outputs over baseline models under F1 metric.

Multimodal Punctuation Prediction with Contextual Dropout

“…We instead use a single prediction for each token, and we find that we can achieve superior performance using much smaller context windows than [1]. Finally, [17,18] apply transformers to punctuation prediction using lexical features and prosodic features which are aligned using pre-trained feature extractors and alignment networks. In contrast to [17,18], we use forced-alignment from ASR and learn acoustic features from scratch from spectrogram segments corresponding each text tokens.…”

Joint prediction of truecasing and punctuation for conversational speech in low-resource scenarios

“…However, the performance of simple n-gram language models suffers when longrange lexical information is required to disambiguate between punctuation classes [10]. Joint modelling of truecasing and punctuation tasks is considered in [11,12] using deep learning models in a classification framework. Authors in [11] assume punctuation as an independent task and truecasing as conditionally dependent on punctuation given latent representation of the input.…”

“…Joint modelling of truecasing and punctuation tasks is considered in [11,12] using deep learning models in a classification framework. Authors in [11] assume punctuation as an independent task and truecasing as conditionally dependent on punctuation given latent representation of the input. However, it is treated as a multi-task problem in [12] where both truecasing and punctuation are independent given the input latent representation.…”

Position-Invariant Truecasing with a Word-and-Character Hierarchical Recurrent Neural Network

“…Word-based truecasing has been the dominant approach for a long time since the introduction of the task by Lita et al (2003). Word-based models can be further categorized into generative models such as HMMs (Lita et al, 2003;Gravano et al, 2009;Beaufays and Strope, 2013;Nebhi et al, 2015) and discriminative models such as Maximum-Entropy Markov Models (Chelba and Acero, 2004), Conditional Random Fields (Wang et al, 2006), and most recently Transformer neural network models (Nguyen et al, 2019;Rei et al, 2020;Sunkara et al, 2020). Word-based models need to refine the class of mixed case words because there is a combinatorial number of possibilities of case mixing for a word (e.g., LaTeX).…”