Punctuation Prediction in Spontaneous Conversations: Can We Mitigate ASR Errors with Retrofitted Word Embeddings?

Augustyniak, Łukasz; Szymański, P.; Morzy, Mikołaj; Żelasko, Piotr; Szymczak, Adrian; Mizgajski, J.; Carmiel, Yishay; Dehak, Najim

doi:10.48550/arxiv.2004.05985

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although it is a well explored problem in the literature, most of these improvements do not directly translate to all domains. In particular, punctuation prediction for conversational speech is not very well explored [16,17,15]. Also, a number of approaches have been proposed exploiting the use of acoustic features in addition to lexical features for punctuation task, but they are rather limited and do not clearly address the gap in performance with ASR outputs.…”

Section: Introductionmentioning

confidence: 99%

Multimodal Semi-supervised Learning Framework for Punctuation Prediction in Conversational Speech

Sunkara¹,

Ronanki²,

Bekal³

et al. 2020

Preprint

View full text Add to dashboard Cite

In this work, we explore a multimodal semi-supervised learning approach for punctuation prediction by learning representations from large amounts of unlabelled audio and text data. Conventional approaches in speech processing typically use forced alignment to encoder per frame acoustic features to word level features and perform multimodal fusion of the resulting acoustic and lexical representations. As an alternative, we explore attention based multimodal fusion and compare its performance with forced alignment based fusion. Experiments conducted on the Fisher corpus show that our proposed approach achieves ∼6-9% and ∼3-4% absolute improvement (F1 score) over the baseline BLSTM model on reference transcripts and ASR outputs respectively. We further improve the model robustness to ASR errors by performing data augmentation with N-best lists which achieves up to an additional ∼2-6% improvement on ASR outputs. We also demonstrate the effectiveness of semi-supervised learning approach by performing ablation study on various sizes of the corpus. When trained on 1 hour of speech and text data, the proposed model achieved ∼9-18% absolute improvement over baseline model.

show abstract

Section: Introductionmentioning

confidence: 99%