Towards End-to-End spoken intent recognition in smart home

Abstract

on clean transcriptions whereas ASR transcriptions contain errors reducing the overall performance. Although the pipeline approach is widely adopted, there is a rising interest for end-toend (E2E) SLU which combines ASR and NLU in one model, avoiding the cumulative ASR and NLU errors of the pipeline approach [2], [3]. The main motivation for applying the E2E approach is that word by word recognition is not needed to infer intents. On top of that, the phoneme dictionary and language model (LM) of the ASR become optional. However, E2E approaches are highly dependent on large training data sets which are difficult to acquire, limiting the applicability to new domains where data is scarce which is the case for smart homes.The main contributions of this paper are: 1) the first work on E2E SLU for voice command in a smart home environment; 2) a comparison of a state-of-the-art pipeline approach that predicts intents from the ASR hypothesis and an E2E SLU model; 3) experiments performed with realistic non-English and synthetic data to deal with the paucity of domain specific data sets. Both approaches are positioned with respect to the state-of-the-art in Section II and are outlined in Section III. We tackle the lack of domain-specific data by using Natural Language Generation (NLG) and text-to-speech (TTS) to generate French voice command training data. An overview of these processes and data sets is given in Sections III and IV. Section V presents the results of experiments on a corpus of real smart home voice commands followed by a discussion, conclusion and outlook on future work. II. RELATED WORKSLU is typically seen as a slot-filling task in order to predict the speaker's intent on the one side and entities in a spoken utterance (slots and values) on the other side [1]. The most common approach is a pipeline of an ASR and an NLU module. The ASR system outputs the hypothesis transcriptions from a speech utterance that are analyzed by the NLU module to extract the meaning. While the slot-filling task is most often Abstract-Voice based interaction in a smart home has become a feature of many industrial products. These systems react to voice commands, whether it is for answering a question, providing music or turning on the lights. To be efficient, these systems must be able to extract the intent of the user from the voice command. Intent recognition from voice is typically performed through automatic speech recognition (ASR) and intent classification from the transcriptions in a pipeline. However, the errors accumulated at the ASR stage might severely impact the intent classifier. I n t his p aper, w e p ropose a n End-to-End (E2E) model to perform intent classification directly from the raw speech input. The E2E approach is thus optimized for this specific task and avoids error propagation. Furthermore, prosodic aspects of the speech signal can be exploited by the E2E model for intent classification (e.g., question vs imperative voice). Experiments on a corpus of voice commands acquired in a real smart home reveal t...

End-to-End Spoken Language Understanding: Performance analyses of a voice command task in a low resource setting

“…On top of that, our correlation tests in Section 5.3 showed that perfect ASR is not necessary to obtain good E2E SLU performance. It is, however, essential in the case of a pipeline approach as we have demonstrated in Desot et al (2019b) for intent prediction and in Desot et al (2019a) for concept prediction. This answers our first question and confirms the state-of-the-art: the E2E model reduces the cascade of error effect.…”

“…As in Desot et al (2019b), the ASR component of our pipeline SLU is the Kaldi tool, nnet2 version. This neural-network ASR training framework allows training with large amounts of data using multiple GPUs or multi-core machines.…”

Low resource end-to-end spoken language understanding with capsule networks

“…Broadly speaking, E2ESLU systems have still not superseded the performance of well-trained multi-step SLU systems. While achieving high accuracies without large amounts of training data is hard in general E2E systems, with additional techniques like pre-training [14], artificial data generation [5] or transfer learning [27], a comparable performance to state-of-the-art ASR-NLU pipeline systems can be achieved with E2ESLU [23,10].…”

Where are we in semantic concept extraction for Spoken Language Understanding?

“…SLU end-to-end systems are usually trained to generate both recognized words and semantic tags [18,15].…”