Punctuation Prediction Model for Conversational Speech

Żelasko, Piotr; Szymański, P.; Mizgajski, J.; Szymczak, Adrian; Carmiel, Yishay; Dehak, Najim

doi:10.48550/arxiv.1807.00543

Cited by 8 publications

(11 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More powerful models have included conditional random fields [4], boosted hierarchical prediction [5], and punctuation as neural machine translation (NMT) [6]. However, state of the art results have been achieved using convolutional neural networks (CNNs) [7], long short term memory (LSTM) networks [8,9,10], and transformers [1,11], treating the problem as a classification task. In these latter approaches, the task is to consider which punctuation symbol should follow each token in an utterance, rather than, say, detecting just sentence boundaries in text.…”

Section: Related Workmentioning

confidence: 99%

“…Various datasets have been proposed for punctuation prediction, including the Fisher corpus [7], TED talks and journalism data [12], and the IWSLT 2012 TED task [1,9,13]. We evaluate our unimodal text-based approach on the IWSLT TED Task dataset, comparing a baseline version of our approach to prior CNN, LSTM, and transformer architectures.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal Punctuation Prediction with Contextual Dropout

Silva

Theobald

Apostoloff

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Automatic speech recognition (ASR) is widely used in consumer electronics. ASR greatly improves the utility and accessibility of technology, but usually the output is only word sequences without punctuation. This can result in ambiguity in inferring user-intent. We first present a transformerbased approach for punctuation prediction that achieves 8% improvement on the IWSLT 2012 TED Task, beating the previous state of the art [1]. We next describe our multimodal model that learns from both text and audio, which achieves 8% improvement over the text-only algorithm on an internal dataset for which we have both the audio and transcriptions. Finally, we present an approach to learning a model using contextual dropout that allows us to handle variable amounts of future context at test time.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Multimodal Punctuation Prediction with Contextual Dropout

Silva

Theobald

Apostoloff

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Some prosodic features like fundamental frequency and energy can be averaged across each word and used as input to the acoustic encoder. Similarly, word duration can also be used as a feature and the work by [16] has shown minor improvements in punctuation prediction for conversational speech by employing relative word timing and duration of word. However, such mechanism does not capture the acoustic context beyond a word and also prevents the use of frame-level acoustic features where the average vector does not represent anything.…”

Section: Forced Alignment Fusionmentioning

confidence: 99%

“…al. [16] with each layer having 128 weights in each direction. We also train another lexical only model which is a pretrained truncated BERT model [31] consisting of 6 transformer self attention layers with each hidden layer of size 768.…”

Section: Model Configurationsmentioning

confidence: 99%

“…Although it is a well explored problem in the literature, most of these improvements do not directly translate to all domains. In particular, punctuation prediction for conversational speech is not very well explored [16,17,15]. Also, a number of approaches have been proposed exploiting the use of acoustic features in addition to lexical features for punctuation task, but they are rather limited and do not clearly address the gap in performance with ASR outputs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal Semi-supervised Learning Framework for Punctuation Prediction in Conversational Speech

Sunkara¹,

Ronanki²,

Bekal³

et al. 2020

Preprint

View full text Add to dashboard Cite

In this work, we explore a multimodal semi-supervised learning approach for punctuation prediction by learning representations from large amounts of unlabelled audio and text data. Conventional approaches in speech processing typically use forced alignment to encoder per frame acoustic features to word level features and perform multimodal fusion of the resulting acoustic and lexical representations. As an alternative, we explore attention based multimodal fusion and compare its performance with forced alignment based fusion. Experiments conducted on the Fisher corpus show that our proposed approach achieves ∼6-9% and ∼3-4% absolute improvement (F1 score) over the baseline BLSTM model on reference transcripts and ASR outputs respectively. We further improve the model robustness to ASR errors by performing data augmentation with N-best lists which achieves up to an additional ∼2-6% improvement on ASR outputs. We also demonstrate the effectiveness of semi-supervised learning approach by performing ablation study on various sizes of the corpus. When trained on 1 hour of speech and text data, the proposed model achieved ∼9-18% absolute improvement over baseline model.

show abstract