Towards Semi-Supervised Semantics Understanding from Speech

Abstract

Much recent work on Spoken Language Understanding (SLU) falls short in at least one of three ways: models were trained on oracle text input and neglected the Automatics Speech Recognition (ASR) outputs, models were trained to predict only intents without the slot values, or models were trained on a large amount of inhouse data. We proposed a clean and general framework to learn semantics directly from speech with semi-supervision from transcribed speech to address these. Our framework is built upon pretrained end-to-end (E2E) ASR and self-supervised language models, such as BERT, and fine-tuned on a limited amount of target SLU corpus. In parallel, we identified two inadequate settings under which SLU models have been tested: noise-robustness and E2E semantics evaluation. We tested the proposed framework under realistic environmental noises and with a new metric, the slots edit F 1 score, on two public SLU corpora. Experiments show that our SLU framework with speech as input can perform on par with those with oracle text as input in semantics understanding, while environmental noises are present, and a limited amount of labeled semantics data is available. * Work performed during an internship at Amazon AI. † Corresponding author. 3 SLU typically consists of Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU). ASR maps audio to text, and NLU maps text to semantics. Here, we are interested in learning a mapping directly from raw audio to semantics. 4 Semantics is commonly formulated as intent and slots in common benchmarking datasets like ATIS.

Improved Spoken Language Representation for Intent Understanding in a Task-Oriented Dialogue System

“…It shows the best performance when speech data itself are used as input, but this end-to-end method requires a lot of computation. The proposed method shows comparable performance to Lai's approach [13] through a simple spoken language representation. As shown in Table 5, the average accuracy of our proposed model achieved the highest performance in the Audio-Snips dataset.…”

“…Chung et al utilized a masking policy approach for the SLU task to jointly pre-train the unpaired speech and text via aligning representations [25]. SpeechBERT was trained via a semi-supervised method, not only representation learning, but also for the intent classification and slot-filling [13]. This model is tested for its demonstration of robustness against ASR errors and extraction of semantic meaning in the input sequence.…”

“…In this section, we analyze the errors of utterances based on recognized text obtained from the recent state-of-the-art ASR model, Wav2vec 2.0 [14]. To this end, we use the FSC [7] dataset and the Audio-Snips [13] (the authors denoted it as "SNIPS-Multi", but it will be referred to in this paper as "Audio-Snips") dataset, which is a widely used SLU benchmark dataset. In the models, which were trained using labeled text and evaluated with labeled and recognized text, we show the issues of the conventional spoken dialogue system that motivate this work through the correlation between the ASR errors and the intent classification performance.…”

“…Table 5 demonstrates the results of the proposed method and the conventional methods. Note that both Huang et al [9] and Cao et al [10] used the recognized text, and Lai et al [13] set the input as speech. For the case using recognized text from the ASR system, the proposed method outperforms other methods.…”

“…In this paper, we focus on exploring and analyzing why the recent SLU models in spoken dialogue situations are still weak at understanding utterances from various users and in various conditions. To this end, for each utterance in two SLU datasets [7,13], we obtain the recognized text utilizing a Wav2vec 2.0 ASR model [14], which is the current state-of-the-art. We then report correlations between both character error rate (CER) and word error rate (WER), and intent classification accuracy, where the higher CER and WER lead the intent classification models to yield low performances.…”