Removing Dimensional Restrictions On Complex/Hyper-Complex Neural Networks

Abstract

It has been shown that the core reasons that complex and hypercomplex valued neural networks offer improvements over their real-valued counterparts is the fact that aspects of their algebra forces treating multi-dimensional data as a single entity. However, both are constrained to a set number of dimensions, two for complex and four for quaternions. These observations motivate us to introduce novel vector map convolutions which capture this property, while dropping the unnatural dimensionality constraints their algebra imposes. This is achieved by introducing a system that mimics the unique linear combination of input dimensions via the Hamilton product using a permutation function, as well as batch normalization and weight initialization for the system. We perform two experiments to show that these novel vector map convolutions seem to capture all the benefits of complex and hyper-complex networks, while avoiding the dimensionality restriction.

Improving Axial-Attention Network Classification via Cross-Channel Weight Sharing

“…The multipliers shown in the figure are "[1, 2, 4, 1]," which gives 26 trainable layers. For the 50-layer version of the network uses the block multipliers " [3,4,6,3]." For our quaternion enhanced model, these layer counts do not include the quaternion modules.…”

“…This experiment increases the layer count of the the standard convolution-based ResNet [5], the quaternionvalued convolution-based ResNet [2], the axial-ResNet [14] from 26 to 35 layers. This was done by using a block multiplier of [2,3,4,2] for these models. Although this did not give us exactly 33 layers, it preserved the bottleneck structure of the original design and enhances comparability.…”

“…The quaternion network learned to accurately reconstruct the colors in unseen images but the real-valued network did not. More recently, the paper [3,4] showed that the properties of the quaternion convolution were due entirely to weight sharing and had no dependence whatsoever on the associated quaternion algebra they were embedded in. Specifically, implementation of the quaternion convolution does not use algebraic operations unique to quaternions.…”

“…These vector-map convolutions can be applied to data of any dimension. All that is needed is a way to choose the vector map coefficients for the higher-dimensional multiplications (which was given in [3,4]). More recently [15] reported a similar idea for the case of fully connected feedforward networks, testing the model using LSTMs and Transformer Networks on various natural language understanding tasks.…”

PHNNs: Lightweight Neural Networks via Parameterized Hypercomplex Convolutions

“…Unfortunately, most common color image datasets contain RGB images and some tricks are required to process this data type with QNNs. Among them, the most used are padding a zero-channel to the input to encapsulate the image in the four quaternion components, or remodeling the QNN layer with the help of vector maps [31]. In addition, while quaternion neural operations are widespread and easy to be integrated in preexisting models, very few attempts have been made to extend models to different domain orders.…”