Learning Asr-Robust Contextualized Embeddings for Spoken Language Understanding

Abstract

Employing pre-trained language models (LM) to extract contextualized word representations has achieved state-of-the-art performance on various NLP tasks. However, applying this technique to noisy transcripts generated by automatic speech recognizer (ASR) is concerned. Therefore, this paper focuses on making contextualized representations more ASRrobust. We propose a novel confusion-aware fine-tuning method to mitigate the impact of ASR errors to pre-trained LMs. Specifically, we fine-tune LMs to produce similar representations for acoustically confusable words that are obtained from word confusion networks (WCNs) produced by ASR. Experiments on the benchmark ATIS dataset show that the proposed method significantly improves the performance of spoken language understanding when performing on ASR transcripts.

St-Bert: Cross-Modal Language Model Pre-Training for End-to-End Spoken Language Understanding

“…For a fair comparison, we exclude the model performance of a data augmentation setting. Note that the experiment settings for Smartlights and Snips datasets are not exactly same as that of Huang and Chen [28] due to the given data condition described in 4.1. Nevertheless, we present both results to show that our model is competitive.…”

“…Previous SLU studies used this dataset by synthesizing audio from text data, and we follow the same experimental protocol. To compare our result with the result of Huang and Chen, we used Google text-to-speech (TTS) system the same as in [28], though it does not guarantee that our data are identical to their audio. SmartLights dataset [29] consists of 1,660 spoken commands for a smart light assistant with 6 unique intents.…”

“…For each utterance, there are two audio types where the microphone setting is different, close field and far field and the latter setting is more challenging. To evaluate our model on this dataset, we use 10-fold cross-validation as suggested in [28].…”

Two-Stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding

“…SNIPS is an NLU benchmark, so they only provide text utterances and their corresponding labels. We generate speech data by Google's commercial speech synthesis toolkit 1 similar to [22] to use SNIPS for SLU evaluation. We use a single speaker option by setting as a basic voice type named en-US-Standard-B.…”

“…We use a single speaker option by setting as a basic voice type named en-US-Standard-B. Because other works [22,23] use their own speech synthesis methods and does not mention exact details to reproduce, a fair comparison between them and ours is impossible.…”