Compressing deep‐quaternion neural networks with targeted regularisation

Abstract

In recent years, hyper‐complex deep networks (such as complex‐valued and quaternion‐valued neural networks – QVNNs) have received a renewed interest in the literature. They find applications in multiple fields, ranging from image reconstruction to 3D audio processing. Similar to their real‐valued counterparts, quaternion neural networks require custom regularisation strategies to avoid overfitting. In addition, for many real‐world applications and embedded implementations, there is the need of designing sufficiently compact networks, with few weights and neurons. However, the problem of regularising and/or sparsifying QVNNs has not been properly addressed in the literature as of now. In this study, the authors show how to address both problems by designing targeted regularisation strategies, which can minimise the number of connections and neurons of the network during training. To this end, they investigate two extensions of ℓ1 and structured regularisations to the quaternion domain. In the authors’ experimental evaluation, they show that these tailored strategies significantly outperform classical (real‐valued) regularisation approaches, resulting in small networks especially suitable for low‐power and real‐time applications.

An Information-Theoretic Perspective on Proper Quaternion Variational Autoencoders

“…The decoder has a mirrored structure, and thus quaternion fully connected layers are piled up as , with an additional refiner layer at the end of the stack. We do not consider including the quaternion batch normalization [ 38 , 45 ] since it can introduce randomness which may affect the correct learning of the distribution statistics [ 24 , 66 ]. For every experiment, the prior distribution is a centered isotropic -proper quaternion Gaussian distribution, as described in Section 4 .…”

“…This properties have been widely exploited in shallow learning models, such as linear and nonlinear adaptive filters [ 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 ]. Another fundamental property of quaternion-valued learning is the Hamilton product, which has recently favored the proliferation of convolutional neural networks in the quaternion domain [ 35 , 36 , 37 , 38 ]. Due to their capabilities, quaternion-valued learning methods have been applied in several applications, including spoken language understanding [ 39 ], color image processing [ 40 , 41 ], 3D audio [ 42 , 43 ], speech recognition [ 44 ], image generation [ 45 ], quantum mechanics [ 46 ], risk diversification [ 47 ], gait data analysis [ 48 ].…”

Quaternion Generative Adversarial Networks

“…However, the covariance matrix is not able to recover the complete second-order statistics in the quaternion domain [4] and the decomposition requires heavy matrix calculations and computational time [18]. Another remarkable approach is introduced in [34], where the input is standardized computing the average of the variance of each component. Nevertheless, describing the second-order statistics of a signal in the quaternion domain needs meticulous computations and the approach in [34] is an approximation of the complete variance.…”

A Quaternion-Valued Variational Autoencoder

“…These advantages are due to the properties of quaternion algebras, including the Hamilton product that is used in quaternion convolutions. This has recently paved the way to the development of novel deep quaternion neural networks [11,13,14], often tailored to specific applications, including theme identification in telephone conversation [15], 3D sound event localization and detection [16,17], heterogeneous image processing [18] and speech recognition [19]. Other properties of quaternion algebra that may be exploited in learning processes are related to the second-order statistics.…”