Quaternion Neural Networks for Spoken Language Understanding

Parcollet, Titouan; Morchid, Mohamed; Bousquet, Pierre-Michel; Dufour, Richard; Linarès, Georges; Mori, Renato De

doi:10.1109/slt.2016.7846290

Cited by 32 publications

(51 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where α is a quaternion split activation function [8,6,9,17]. The output layer of a quaternion neural network is commonly either quaternion-valued such as for quaternion approximation [7], or real-valued to obtains a posterior distribution based on a softmax function following the split approach.…”

Section: Quaternion Convolutional Encoder-decodermentioning

confidence: 99%

Quaternion Convolutional Neural Networks for Heterogeneous Image Processing

Parcollet¹,

Morchid²,

Linarès³

2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Convolutional neural networks (CNN) have recently achieved state-of-the-art results in various applications. In the case of image recognition, an ideal model has to learn independently of the training data, both local dependencies between the three components (R,G,B) of a pixel, and the global relations describing edges or shapes, making it efficient with small or heterogeneous datasets. Quaternion-valued convolutional neural networks (QCNN) solved this problematic by introducing multidimensional algebra to CNN. This paper proposes to explore the fundamental reason of the success of QCNN over CNN, by investigating the impact of the Hamilton product on a color image reconstruction task performed from a gray-scale only training. By learning independently both internal and external relations and with less parameters than real valued convolutional encoder-decoder (CAE), quaternion convolutional encoder-decoders (QCAE) perfectly reconstructed unseen color images while CAE produced worst and gray-scale versions.Index Terms-Quaternion convolutional encoder-decoder, convolutional neural networks, heterogeneous image processing

show abstract

Section: Quaternion Convolutional Encoder-decodermentioning

confidence: 99%

Quaternion Convolutional Neural Networks for Heterogeneous Image Processing

Parcollet¹,

Morchid²,

Linarès³

2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…with σ and α the sigmoid and tanh quaternion split activations [18,11,19,10]. The quaternion weight and bias matrices are initialized following the proposal of [15].…”

Section: Quaternion Long-short Term Memory Neural Networkmentioning

confidence: 99%

Bidirectional Quaternion Long Short-term Memory Recurrent Neural Networks for Speech Recognition

Parcollet

Morchid

Linarès

et al. 2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Recurrent neural networks (RNN) are at the core of modern automatic speech recognition (ASR) systems. In particular, long-short term memory (LSTM) recurrent neural networks have achieved state-of-the-art results in many speech recognition tasks, due to their efficient representation of long and short term dependencies in sequences of inter-dependent features. Nonetheless, internal dependencies within the element composing multidimensional features are weakly considered by traditional real-valued representations. We propose a novel quaternion long-short term memory (QL-STM) recurrent neural network that takes into account both the external relations between the features composing a sequence, and these internal latent structural dependencies with the quaternion algebra. QLSTMs are compared to LSTMs during a memory copy-task and a realistic application of speech recognition on the Wall Street Journal (WSJ) dataset. QLSTM reaches better performances during the two experiments with up to 2.8 times less learning parameters, leading to a more expressive representation of the information.

show abstract

“…Moreover, it performs an interpolation between two rotations following a geodesic over a sphere in the R 3 space. 1 A multilayer perceptron (MLP) with more than one hidden layer Given a segmentation S = {s 1 , s 2 , s 3 , s 4 } of a document p ∈ P depending on the document segmentation detailed in [12] and a set of topics from a latent Dirichlet allocation (LDA) [23] z = {z 1 , . .…”

Section: Quaternion Algebramentioning

confidence: 99%

“…In other words, real-valued representations reveal little in way of document internal structure by only considering words or topics contained in the document as an isolate basic element. Therefore, quaternion multi-layer perceptrons (QMLP) [11,12,13] and quaternion autoencoders (QAE) [14] have been introduced to capture such latent dependencies, thanks to the fourth dimensionality of hyper-complex numbers alongside to the Hamilton product [15]. Nonetheless, previous quaternion-based studies focused on three-layered neural networks, while the efficiency and the effectiveness of DNN have already been demonstrated [16,5].…”

Section: Introductionmentioning

confidence: 99%

Deep quaternion neural networks for spoken language understanding

Parcollet

Morchid

Linarès

2017

2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

Self Cite

View full text Add to dashboard Cite

To cite this version:Titouan Parcollet, Mohamed Morchid, Georges Linarès. Deep quaternion neural networks for spoken language understanding. ABSTRACTDeep Neural Networks (DNN) received a great interest from researchers due to their capability to construct robust abstract representations of heterogeneous documents in a latent subspace. Nonetheless, mere real-valued deep neural networks require an appropriate adaptation, such as the convolution process, to capture latent relations between input features. Moreover, real-valued deep neural networks reveal little in way of document internal dependencies, by only considering words or topics contained in the document as an isolate basic element. Quaternion-valued multi-layer perceptrons (QMLP), and autoencoders (QAE) have been introduced to capture such latent dependencies, alongside to represent multidimensional data. Nonetheless, a three-layered neural network does not benefit from the high abstraction capability of DNNs. The paper proposes first to extend the hyper-complex algebra to deep neural networks (QDNN) and, then, introduces pre-trained deep quaternion neural networks (QDNN-AE) with dedicated quaternion encoder-decoders (QAE). The experiments conduced on a theme identification task of spoken dialogues from the DECODA data set show, inter alia, that the QDNN-AE reaches a promising gain of 2.2% compared to the standard real-valued DNN-AE.Index Terms-Quaternions, deep neural networks, spoken language understanding, autoencoders, machine learning.

show abstract

Quaternion Neural Networks for Spoken Language Understanding

Cited by 32 publications

References 8 publications

Quaternion Convolutional Neural Networks for Heterogeneous Image Processing

Quaternion Convolutional Neural Networks for Heterogeneous Image Processing

Bidirectional Quaternion Long Short-term Memory Recurrent Neural Networks for Speech Recognition

Deep quaternion neural networks for spoken language understanding

Contact Info

Product

Resources

About