Distilling Knowledge from an Ensemble of Models for Punctuation Prediction

Yi, Jiangyan; Tao, Jianhua; Wen, Zhengqi; Li, Ya

doi:10.21437/interspeech.2017-1079

Cited by 28 publications

(34 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In another study, Gale and Parthasarathy (2017) used character-level LSTM architecture to achieve results that are competitive with the wordlevel CRF based approach. Yi et al (2017) combined bidirectional LSTM with a CRF layer and an ensemble of three networks. They further used knowledge distillation to transfer knowledge from the ensemble of networks to a single DNN network.…”

Section: Related Workmentioning

confidence: 99%

Punctuation Restoration using Transformer Models for High-and Low-Resource Languages

Alam¹,

Khan²,

Alam³

2020

Proceedings of the Sixth Workshop on Noisy User-Generated Text (W-Nut 2020)

View full text Add to dashboard Cite

Punctuation restoration is a common postprocessing problem for Automatic Speech Recognition (ASR) systems. It is important to improve the readability of the transcribed text for the human reader and facilitate NLP tasks. Current state-of-art address this problem using different deep learning models. Recently, transformer models have proven their success in downstream NLP tasks, and these models have been explored very little for the punctuation restoration problem. In this work, we explore different transformer based models and propose an augmentation strategy for this task, focusing on high-resource (English) and low-resource (Bangla) languages. For English, we obtain comparable state-of-the-art results, while for Bangla, it is the first reported work, which can serve as a strong baseline for future work. We have made our developed Bangla dataset publicly available for the research community.

show abstract

Section: Related Workmentioning

confidence: 99%

Punctuation Restoration using Transformer Models for High-and Low-Resource Languages

Alam¹,

Khan²,

Alam³

2020

Proceedings of the Sixth Workshop on Noisy User-Generated Text (W-Nut 2020)

View full text Add to dashboard Cite

show abstract

“…Full-Transformer denotes the model that replaces the CT-Transformer block in Figure 1 with a full sequence Transformer. The results of these models on the IWSLT2011 test set are reported in the last group in [19], respectively. The previous stateof-the-art model is Self-attention-word-speech [2], which used a full sequence Transformer encoder-decoder model with pre-trained word2vec and speech2vec embedding features.…”

Section: Resultsmentioning

confidence: 99%

Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detection

Qian

Chen

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

With the increased applications of automatic speech recognition (ASR) in recent years, it is essential to automatically insert punctuation marks and remove disfluencies in transcripts, to improve the readability of the transcripts as well as the performance of subsequent applications, such as machine translation, dialogue systems, and so forth. In this paper, we propose a Controllable Time-delay Transformer (CT-Transformer) model that jointly completes the punctuation prediction and disfluency detection tasks in real time. The CT-Transformer model facilitates freezing partial outputs with controllable time delay to fulfill the real-time constraints in partial decoding required by subsequent applications. We further propose a fast decoding strategy to minimize latency while maintaining competitive performance. Experimental results on the IWSLT2011 benchmark dataset and an in-house Chinese annotated dataset demonstrate that the proposed approach outperforms the previous state-of-the-art models on F-scores and achieves a competitive inference speed.

show abstract

“…Though non-sequential, several previous approaches use simpler network architectures (e.g. DNNs (Yi et al, 2017;Che et al, 2016) or CNNs (B. Garg and Anika, 2018;Che et al, 2016;Żelasko et al, 2018)), which have less predictive power.…”

Section: Related Workmentioning

confidence: 99%

Proceedings of the 17th International Conference on Spoken Language Translation

Federico¹,

Waibel²,

Knight³

et al. 2020

View full text Add to dashboard Cite

The conference chairs and organizers would like to express their gratitude to everyone who contributed and supported IWSLT. Our IWSLT-20 program exceeds all our expectations in quality and breath, particularly when considering the challenges during a pandemic under lock-downs and health and travel restrictions. We thank the challenge track chairs, organizers, and participants, the program chairs and committee members, as well as all the authors that went the extra mile to submit system and research papers to IWSLT, and make this year's conference our most vibrant than ever. We also wish to express our sincere gratitude to ACL for hosting our conference and for arranging the logistics and infrastructure that allow us to hold IWSLT 2020 as a virtual online conference.

show abstract

Distilling Knowledge from an Ensemble of Models for Punctuation Prediction

Cited by 28 publications

References 22 publications

Punctuation Restoration using Transformer Models for High-and Low-Resource Languages

Punctuation Restoration using Transformer Models for High-and Low-Resource Languages

Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detection

Proceedings of the 17th International Conference on Spoken Language Translation

Contact Info

Product

Resources

About