Style Attuned Pre-training and Parameter Efficient Fine-tuning for Spoken Language Understanding

Cao, Jin; Wang, Jun; Hamza, Wael; Vanee, Kelly; Li, Shang-Wen

doi:10.48550/arxiv.2010.04355

Cited by 5 publications

(12 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their scheme reduced the similar semantics for phonology of hardly distinguishable words with fine-tuning. Cao et al showed the pre-trained method for conversational language modeling that enabled the SLU networks to catch the linguistic representations in dialogue styles with the ASR errors [10]. Chung et al utilized a masking policy approach for the SLU task to jointly pre-train the unpaired speech and text via aligning representations [25].…”

Section: Asr-slu-based Intent Classificationmentioning

confidence: 99%

“…Table 5 demonstrates the results of the proposed method and the conventional methods. Note that both Huang et al [9] and Cao et al [10] used the recognized text, and Lai et al [13] set the input as speech. For the case using recognized text from the ASR system, the proposed method outperforms other methods.…”

Section: Comparisons To Other Studiesmentioning

confidence: 99%

“…Both mentioned methods were applied to S and evaluated on Ŝ. In addition, the experiment was conducted with a single BERT model, and the two methods are denoted as S Mask [10,20,30] → Ŝ and S Mix → Ŝ, respectively.…”

Section: Ablation Studiesmentioning

confidence: 99%

“…To address the aforementioned problem, several ASR-SLU-based studies [7][8][9][10][11][12] have strived to understand users' intents based on jointly training with both speech and corresponding text (labeled text). To improve the users' intents classification, however, these approaches are still vulnerable to the faults in the recognized text obtained by the ASR models.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Improved Spoken Language Representation for Intent Understanding in a Task-Oriented Dialogue System

Kim

Yoon

Jung

2022

Sensors

View full text Add to dashboard Cite

Successful applications of deep learning technologies in the natural language processing domain have improved text-based intent classifications. However, in practical spoken dialogue applications, the users’ articulation styles and background noises cause automatic speech recognition (ASR) errors, and these may lead language models to misclassify users’ intents. To overcome the limited performance of the intent classification task in the spoken dialogue system, we propose a novel approach that jointly uses both recognized text obtained by the ASR model and a given labeled text. In the evaluation phase, only the fine-tuned recognized language model (RLM) is used. The experimental results show that the proposed scheme is effective at classifying intents in the spoken dialogue system containing ASR errors.

show abstract

Section: Asr-slu-based Intent Classificationmentioning

confidence: 99%

Section: Comparisons To Other Studiesmentioning

confidence: 99%

Section: Ablation Studiesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Improved Spoken Language Representation for Intent Understanding in a Task-Oriented Dialogue System

Kim

Yoon

Jung

2022

Sensors

View full text Add to dashboard Cite

show abstract

“…The key contributions of this paper are summarized as follows: • We introduce a semi-supervised SLU framework for learning semantics from speech to alleviate: (1) the need for a large amount of in-house, homogenous data [2,7,8,17], (2) the limitation of only intent classification [8,9,13] by predicting text, slots and intents, and (3) any additional manipulation on labels or loss, such as label projection [26], output serialization [7,18,19], ASR n-best hypothesis, or ASR-robust training losses [13,27]. Figure 2 illustrates our approach.…”

Section: Introductionmentioning

confidence: 99%

Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining

Lai¹,

Chuang

Lee

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Much recent work on Spoken Language Understanding (SLU) is limited in at least one of three ways: models were trained on oracle text input and neglected ASR errors, models were trained to predict only intents without the slot values, or models were trained on a large amount of in-house data. In this paper, we propose a clean and general framework to learn semantics directly from speech with semi-supervision from transcribed or untranscribed speech to address these issues. Our framework is built upon pretrained end-toend (E2E) ASR and self-supervised language models, such as BERT, and fine-tuned on a limited amount of target SLU data. We study two semi-supervised settings for the ASR component: supervised pretraining on transcribed speech, and unsupervised pretraining by replacing the ASR encoder with self-supervised speech representations, such as wav2vec. In parallel, we identify two essential criteria for evaluating SLU models: environmental noise-robustness and E2E semantics evaluation. Experiments on ATIS show that our SLU framework with speech as input can perform on par with those using oracle text as input in semantics understanding, even though environmental noise is present and a limited amount of labeled semantics data is available for training.

show abstract

Two-stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding

Kim

Shin

et al. 2020

Preprint

View full text Add to dashboard Cite

End-to-end approaches open a new way for more accurate and efficient spoken language understanding (SLU) systems by alleviating the drawbacks of traditional pipeline systems. Previous works exploit textual information for an SLU model via pre-training with automatic speech recognition or finetuning with knowledge distillation. To utilize textual information more effectively, this work proposes a two-stage textual knowledge distillation method that matches utterancelevel representations and predicted logits of two modalities during pre-training and fine-tuning, sequentially. We use vq-wav2vec BERT as a speech encoder because it captures general and rich features. Furthermore, we improve the performance, especially in a low-resource scenario, with data augmentation methods by randomly masking spans of discrete audio tokens and contextualized hidden representations. Consequently, we push the state-of-the-art on the Fluent Speech Commands, achieving 99.7% test accuracy in the full dataset setting and 99.5% in the 10% subset setting. Throughout the ablation studies, we empirically verify that all used methods are crucial to the final performance, providing the best practice for spoken language understanding. Code to reproduce our results will be available upon publication.

show abstract

Style Attuned Pre-training and Parameter Efficient Fine-tuning for Spoken Language Understanding

Cited by 5 publications

References 13 publications

Improved Spoken Language Representation for Intent Understanding in a Task-Oriented Dialogue System

Improved Spoken Language Representation for Intent Understanding in a Task-Oriented Dialogue System

Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining

Two-stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding

Contact Info

Product

Resources

About