Low resource end-to-end spoken language understanding with capsule networks

Abstract

Designing a Spoken Language Understanding (SLU) system for command-and-control applications is challenging. Both Automatic Speech Recognition and Natural Language Understanding are language and application dependent to a great extent. Even with a lot of design effort, users often still have to know what to say to the system for it to do what they want. We propose to use an end-to-end SLU system that maps speech directly to semantics and that can be trained by the user through demonstrations. The user can teach the system a new command by uttering the command and subsequently demonstrating its meaning through an alternative interface. The system will learn the mapping from the spoken command to the task. The dependency on the user also allows different languages and non-standard or impaired speech as valid inputs. Teaching the system requires effort from the user, so it is crucial that the system learns quickly. In this paper we propose to use capsule networks for this task, which are believed to be data efficient. We discuss two architectures for using capsule networks. We analyse their performance and compare them with two baseline systems, one based on Non-negative Matrix Factorisation (NMF) which has been successful for this task and one encoder-decoder approach. We show that in most cases the capsule network performs better than the baseline systems. Furthermore, we demonstrate the versatility of the architecture by inferring speaker identity and the user's word choice through multitask learning.

Emotion Recognition with Capsule Neural Network

“…The authors showed that the end-to-end speech recognition system with capsule networks on one-second speech commands dataset achieves better results on both clean and noise-added tests than baseline convolutional neural network models. A capsule network for low resource spoken language understanding was proposed for commandand-control applications in [5]. For small quantities of data, the proposed model is shown to significantly outperform the previous state-of-the-art model.…”

Speech to Text by Using the Sindhi Language

“…According to the survey results, the shortage of publicly available continuous speech corpora justifies additional study focus in this field. It also illustrates the need of big corpora or a benchmark in boosting Arabic language research for good human-computer interaction.A Sindhi Unicode-8-based linguistics data set is also multi-class and multi-featured [19]. It was created to address natural language processing (NLP) and linguistic issues in the Sindhi language.…”

The Investigation of Different Loss Functions with Capsule Networks for Speech Emotion Recognition

“…e margin loss was replaced by the computation of connectionist temporal classification (CTC). In another paper, Poncelet et al [10] used capsule networks with recurrent neural networks, additionally encoding time information-an essential property present in speech. ey applied this approach in the field of spoken language understanding (SLU).…”

“…One important consideration in performing deep learning or machine learning in general is the choice of the loss function. Most of the capsule network implementations in other literature [9][10][11] use the original margin loss as described by Sabour et al [8]. Only a few have attempted deviating from the original implementation and instead have employed other loss functions.…”