A Light Transformer For Speech-To-Intent Applications

Abstract

Spoken language understanding (SLU) systems can make life more agreeable, safer (e.g. in a car) or can increase the independence of physically challenged users. However, due to the many sources of variation in speech, a well-trained system is hard to transfer to other conditions like a different language or to speech impaired users. A remedy is to design a user-taught SLU system that can learn fully from scratch from users' demonstrations, which in turn requires that the system's model quickly converges after only a few training samples. In this paper, we propose a light transformer structure by using a simplified relative position encoding with the goal to reduce the model size and improve efficiency. The light transformer works as an alternative speech encoder for an existing user-taught multitask SLU system. Experimental results on three datasets with challenging speech conditions prove our approach outperforms the existed system and other state-of-art models with half of the original model size and training time.

Bottleneck Low-rank Transformers for Low-resource Spoken Language Understanding

“…The light transformer is a light version of the vanilla transformer which includes a low-dimensional (light) relative position encoding (PE) matrix [23]. We will shortly recap vanilla transformers first [24].…”

“…To account for order information, the vanilla transformer will add a d-dimensional absolute PE to the content embedding x which requires the network to learn in which subspace relevant data variation occurs and in which subspace position is represented. To avoid the hassle brought by the additive highdimensional PE, we introduced 6-dimensional relative position encoding in [23]. It is defined by Eq.…”

“…The well-trained model intrinsically holds a low-rank property without degradation of performance. The low-rank design is applied to the light transformer [23] in a SLU context with very scarce training data. The light transformer is a compact version of the vanilla transformer [24].…”

“…The light transformer is a compact version of the vanilla transformer [24]. [23] shows a successful implementation of the compact low-resource SLU system built with the light transformer combined with the capsule network which has only 1.3M parameters but achieves 98.8% accuracy on FluentSpeech Command data without pre-training.…”

A Study into Pre-Training Strategies for Spoken Language Understanding on Dysarthric Speech

“…Training data is very scarce for several reasons including the increased effort required to collect data from this population [4,5]. [6,7,8] present a remedy for low resource SLU by designing a user-taught SLU system. "User-taught" refers to the strategy in which the SLU agent learns from scratch only based on spoken commands and corresponding task demonstrations from its users.…”