Semantic Mask for Transformer Based End-to-End Speech Recognition

Wang, Chengyi; Wu, Yu; Du, Yujiao; Li, Jinyu; Liu, Shujie; Lu, Liang; Ren, Shuo; Ye, Guoli; Zhao, Shenghui; Zhou, Ming

doi:10.21437/interspeech.2020-1778

Cited by 28 publications

(39 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The design of the Frame-based Masked Language Model task is inspired by the Masked Language Model (MLM) objective of BERT (Devlin et al, 2019) and semantic mask for ASR task (Wang et al, 2019a). This task enables the encoder to understand the inner meaning of a segment of speech.…”

Section: Frame-based Masked Language Modelmentioning

confidence: 99%

Curriculum Pre-training for End-to-End Speech Translation

Wang

Liu

et al. 2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

End-to-end speech translation poses a heavy burden on the encoder because it has to transcribe, understand, and learn cross-lingual semantics simultaneously. To obtain a powerful encoder, traditional methods pre-train it on ASR data to capture speech features. However, we argue that pre-training the encoder only through simple speech recognition is not enough, and high-level linguistic knowledge should be considered. Inspired by this, we propose a curriculum pre-training method that includes an elementary course for transcription learning and two advanced courses for understanding the utterance and mapping words in two languages. The difficulty of these courses is gradually increasing. Experiments show that our curriculum pre-training method leads to significant improvements on En-De and En-Fr speech translation benchmarks. * Works are done during internship at Microsoft (a) previous encoder pre-training (b) curriculum encoder pre-training

show abstract

Section: Frame-based Masked Language Modelmentioning

confidence: 99%

Curriculum Pre-training for End-to-End Speech Translation

Wang

Liu

et al. 2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…As with the utterance-wise evaluation, the Early Exit Transformer mod-Table 1: Utterance-wise evaluation. Two numbers in a cell denote %WER of the hybrid SR model used in LibriCSS [18] and end-to-end transformer based SR model [16]. 0S: 0% overlap with short inter-utterance silence.…”

Section: Evaluation Resultsmentioning

confidence: 99%

“…One is a hybrid system with a BLSTM based acoustic model and a 4-gram language model as used in the original LibriCSS paper [18]. The other is one of the best open-source endto-end transformer [16] based ASR models 1 which achieves WERs of 2.08% and 4.95% for LibriSpeech test-clean and test-other, respectively. As with [18], by leveraging the multiple microphones, the individual target signals are generated with mask-based adaptive minimum variance distortionless response (MVDR) beamforming.…”

Section: Implementation Detailsmentioning

confidence: 99%

See 1 more Smart Citation

Don’t Shoot Butterfly with Rifles: Multi-Channel Continuous Speech Separation with Early Exit Transformer

Chen

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

With its strong modeling capacity that comes from a multi-head and multi-layer structure, Transformer is a very powerful model for learning a sequential representation and has been successfully applied to speech separation recently. However, multi-channel speech separation sometimes does not necessarily need such a heavy structure for all time frames especially when the cross-talker challenge happens only occasionally. For example, in conversation scenarios, most regions contain only a single active speaker, where the separation task downgrades to a single speaker enhancement problem. It turns out that using a very deep network structure for dealing with signals with a low overlap ratio not only negatively affects the inference efficiency but also hurts the separation performance. To deal with this problem, we propose an early exit mechanism, which enables the Transformer model to handle different cases with adaptive depth. Experimental results indicate that not only does the early exit mechanism accelerate the inference, but it also improves the accuracy.

show abstract

“…The proposed approach, referred to as Top-Down SLU (TD-SLU), is based on Top-down Attention, a technique inspired from Cognitive Science [8,9] and already successfully applied in the computer vision [10,11] domain. More specifically, on top of the traditional transformer Sequence-to-Sequence (s2s) model [12] used in recent E2E SLU models [13,14], we introduce an additional transformer encoder for the NLU task, which allows the model to utilize the entire sequence to gain semantic understanding by attention. In addition, following the multimodal literature, we combine the top-level NLU feature with the low-level acoustic features using an attention gate [15], which generates a shift in the low-level representation, adapting it according to the high-level information, thus improving the performance.…”

Section: Introductionmentioning

confidence: 99%

Top-Down Attention in End-to-End Spoken Language Understanding

Chen

Mottini

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Spoken language understanding (SLU) is the task of inferring the semantics of spoken utterances. Traditionally, this has been achieved with a cascading combination of Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) modules that are optimized separately, which can lead to a suboptimal overall performance. More recently, End-to-End SLU (E2E SLU) was proposed to perform SLU directly from speech through a joint optimization of the modules, addressing some of the traditional SLU shortcomings.A key challenge of this approach is how to best integrate the feature learning of the ASR and NLU sub-tasks to maximize their performance. While it is known that in general, ASR models focus on low-level features, and NLU models need higher-level contextual information, ASR models can nonetheless also leverage top-down syntactic and semantic information to improve their recognition. Based on this insight, we propose Top-Down SLU (TD-SLU), a new transformer-based E2E SLU model that uses top-down attention and an attention gate to fuse high-level NLU features with low-level ASR features, which leads to a better optimization of both tasks. We have validated our model using the public FluentSpeech set, and a large custom dataset. Results show TD-SLU is able to outperform selected baselines both in terms of ASR and NLU quality metrics, and suggest that the added syntactic and semantic high-level information can improve the model's performance.

show abstract

Semantic Mask for Transformer Based End-to-End Speech Recognition

Cited by 28 publications

References 0 publications

Curriculum Pre-training for End-to-End Speech Translation

Curriculum Pre-training for End-to-End Speech Translation

Don’t Shoot Butterfly with Rifles: Multi-Channel Continuous Speech Separation with Early Exit Transformer

Top-Down Attention in End-to-End Spoken Language Understanding

Contact Info

Product

Resources

About