Deep quaternion neural networks for spoken language understanding

Abstract

To cite this version:Titouan Parcollet, Mohamed Morchid, Georges Linarès. Deep quaternion neural networks for spoken language understanding. ABSTRACTDeep Neural Networks (DNN) received a great interest from researchers due to their capability to construct robust abstract representations of heterogeneous documents in a latent subspace. Nonetheless, mere real-valued deep neural networks require an appropriate adaptation, such as the convolution process, to capture latent relations between input features. Moreover, real-valued deep neural networks reveal little in way of document internal dependencies, by only considering words or topics contained in the document as an isolate basic element. Quaternion-valued multi-layer perceptrons (QMLP), and autoencoders (QAE) have been introduced to capture such latent dependencies, alongside to represent multidimensional data. Nonetheless, a three-layered neural network does not benefit from the high abstraction capability of DNNs. The paper proposes first to extend the hyper-complex algebra to deep neural networks (QDNN) and, then, introduces pre-trained deep quaternion neural networks (QDNN-AE) with dedicated quaternion encoder-decoders (QAE). The experiments conduced on a theme identification task of spoken dialogues from the DECODA data set show, inter alia, that the QDNN-AE reaches a promising gain of 2.2% compared to the standard real-valued DNN-AE.Index Terms-Quaternions, deep neural networks, spoken language understanding, autoencoders, machine learning.

Neural Networks for Natural Language Processing

“…More recently neural networks of complex and hypercomplex numbers have received an increasing attention (Hirose et Yoshida, 2012;Tygert et al, 2016;Danihelka et al, 2016;Wisdom et al, 2016) and some efforts have shown promising results in different applications. In particular a deep quaternion network (Parcollet et al, 2016;Titouan et al, 2017;Parcollet et al, 2017), a deep quaternion convolutional network (Chase Gaudet, 2017;Titouan et al, 2018) and a quaternion recurrent neural network (Parcollet et al, 2018c) have been developed and employed by out team from the LIA for different challenging tasks such as images classification, compression and reconstruction or speech recognition.…”

“…Consequently α = 50 T , with T the number of topics, and β = 0.01. The number T has been previously investigated for this task in (Parcollet et al, 2016(Parcollet et al, , 2017, and is set to 25. More precisely, 10 runs of the T = 25 LDA model are concatenated to obtain a final vector of size 25 × 10 = 250 to alleviate any variations.…”

“…During the last 5 years, we have proposed to remedying this weaknesses by extending the complex features based on Quaternion algebra presented in (Morchid et al, 2013) to neural networks. The effectiveness of such as algebra to represent multidimensional objects have been studied for speech recognition (Parcollet et al, 2018c(Parcollet et al, ,d,a, 2019Titouan et al, 2019b), spoken language understanding (Parcollet et al, 2017;Titouan et al, 2017;Parcollet et al, 2016) and image processing ; moreover, different quaternion based neural networks have been studied such as multi-layer perceptron (Parcollet et al, 2016(Parcollet et al, , 2017, encoder-decoders (Titouan et al, 2017), recurrent neural networks (Parcollet et al, 2018c and convolutional neural networks called QCNN (Parcollet et al, 2018d. One can refer to these papers for further details.…”

Bidirectional Quaternion Long Short-term Memory Recurrent Neural Networks for Speech Recognition

“…with σ and α the sigmoid and tanh quaternion split activations [18,11,19,10]. The quaternion weight and bias matrices are initialized following the proposal of [15].…”

An Information-Theoretic Perspective on Proper Quaternion Variational Autoencoders

“…Another fundamental property of quaternion-valued learning is the Hamilton product, which has recently favored the proliferation of convolutional neural networks in the quaternion domain [ 35 , 36 , 37 , 38 ]. Due to their capabilities, quaternion-valued learning methods have been applied in several applications, including spoken language understanding [ 39 ], color image processing [ 40 , 41 ], 3D audio [ 42 , 43 ], speech recognition [ 44 ], image generation [ 45 ], quantum mechanics [ 46 ], risk diversification [ 47 ], gait data analysis [ 48 ].…”