Integration of Pre-Trained Networks with Continuous Token Interface for End-to-End Spoken Language Understanding

Seo, Seunghyun; Kwak, Donghyun; Lee, Bowon

doi:10.1109/icassp43922.2022.9747047

Cited by 12 publications

(16 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We observed a slight but clear gain by increasing K, which improved both ASR and SLU performance thanks to BERT. We note that our result outperforms the state-of-the-art 86.9% reported in (Seo et al, 2022).…”

Section: Resultscontrasting

confidence: 40%

BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model

Higuchi¹,

Yan²,

Arora³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper presents BERT-CTC, a novel formulation of end-to-end speech recognition that adapts BERT for connectionist temporal classification (CTC). Our formulation relaxes the conditional independence assumptions used in conventional CTC and incorporates linguistic knowledge through the explicit output dependency obtained by BERT contextual embedding. BERT-CTC attends to the full contexts of the input and hypothesized output sequences via the self-attention mechanism. This mechanism encourages a model to learn inner/inter-dependencies between the audio and token representations while maintaining CTC's training efficiency. During inference, BERT-CTC combines a mask-predict algorithm with CTC decoding, which iteratively refines an output sequence. The experimental results reveal that BERT-CTC improves over conventional approaches across variations in speaking styles and languages. Finally, we show that the semantic representations in BERT-CTC are beneficial towards downstream spoken language understanding tasks.

show abstract

Section: Resultscontrasting

confidence: 40%

BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model

Higuchi¹,

Yan²,

Arora³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Trained on text NLU* (Bastianelli et al, 2020) 84.84 -NLU+ (Seo et al, 2021) 87.73 84.34 BART (Lewis et al, 2020) 88 Trained on text Attention BiRNN (Liu and Lane, 2016) 91.10 94.20 Capsule-NLU (Zhang et al, 2019) 95.00 95.20 LIDSNet (Agarwal et al, 2021) 95.97 -SF-ID Network (E et al, 2019) 96.60 95.60 SyntacticTF 97.31 96.01 BERT SLU 97.50 96.10 Stack-Prop. (Qin et al, 2019) 96.90 95.90 Stack-Prop.…”

Section: Modelsmentioning

confidence: 99%

MTL-SLT: Multi-Task Learning for Spoken Language Tasks

Huang¹,

Mv²,

Raju³

et al. 2022

Proceedings of the 4th Workshop on NLP for Conversational AI

View full text Add to dashboard Cite

Language understanding in speech-based systems has attracted extensive interest from both academic and industrial communities in recent years with the growing demand for voice-based applications. Prior works focus on independent research by the automatic speech recognition (ASR) and natural language processing (NLP) communities, or on jointly modeling the speech and NLP problems focusing on a single dataset or single NLP task. To facilitate the development of spoken language research, we introduce MTL-SLT, a multi-task learning framework for spoken language tasks. MTL-SLT takes speech as input, and outputs transcription, intent, named entities, summaries, and answers to text queries, supporting the tasks of spoken language understanding, spoken summarization and spoken question answering respectively. The proposed framework benefits from three key aspects: 1) pre-trained sub-networks of ASR model and language model; 2) multitask learning objective to exploit shared knowledge from different tasks; 3) end-to-end training of ASR and downstream NLP task based on sequence loss. We obtain state-of-the-art results on spoken language understanding tasks such as SLURP and ATIS. Spoken summarization results are reported on a new dataset: Spoken-Gigaword.

show abstract

“…Recently, there is growing interest in building E2E SLU models where the acoustic and textual models are jointly optimized, leading to more robust SLU models [5][6][7][8][9][10][11][12][13][14][15][16]. Early works [5,6] learn an utterance-level semantic representation directly from audio signals without performing speech recognition.…”

Section: Introductionmentioning

confidence: 99%

“…One way to achieve E2E training is to re-frame SLU as a sequence-to-sequence task, where semantic labels are treated as another sequence of output labels besides the transcript [9][10][11][12]. Another way is to unify ASR and NLU models and train them together via differentiable neural interfaces [13][14][15][16]. One commonly used neural interface is to feed the token level hidden representations from ASR as input to the NLU model [13][14][15][16].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

End-to-End Spoken Language Understanding Using Joint CTC Loss and Self-Supervised, Pretrained Acoustic Encoders

Wang

Radfar

Wei

et al. 2023

ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

It is challenging to extract semantic meanings directly from audio signals in spoken language understanding (SLU), due to the lack of textual information. Popular end-to-end (E2E) SLU models utilize sequence-to-sequence automatic speech recognition (ASR) models to extract textual embeddings as input to infer semantics, which, however, require computationally expensive auto-regressive decoding. In this work, we leverage self-supervised acoustic encoders fine-tuned with Connectionist Temporal Classification (CTC) to extract textual embeddings and use joint CTC and SLU losses for utterance-level SLU tasks. Experiments show that our model achieves 4% absolute improvement over the the state-of-theart (SOTA) dialogue act classification model on the DSTC2 dataset and 1.3% absolute improvement over the SOTA SLU model on the SLURP dataset.

show abstract

Integration of Pre-Trained Networks with Continuous Token Interface for End-to-End Spoken Language Understanding

Cited by 12 publications

References 15 publications

BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model

BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model

MTL-SLT: Multi-Task Learning for Spoken Language Tasks

End-to-End Spoken Language Understanding Using Joint CTC Loss and Self-Supervised, Pretrained Acoustic Encoders

Contact Info

Product

Resources

About