All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation

Abstract

Deep neural network is difficult to train and this predicament becomes worse as the depth increases. The essence of this problem exists in the magnitude of backpropagated errors that will result in gradient vanishing or exploding phenomenon. We show that a variant of regularizer which utilizes orthonormality among different filter banks can alleviate this problem. Moreover, we design a backward error modulation mechanism based on the quasiisometry assumption between two consecutive parametric layers. Equipped with these two ingredients, we propose several novel optimization solutions that can be utilized for training a specific-structured (repetitively triple modules of Conv-BN-ReLU) extremely deep convolutional neural network (CNN) WITHOUT any shortcuts/ identity mappings from scratch. Experiments show that our proposed solutions can achieve distinct improvements for a 44-layer and a 110-layer plain networks on both the CIFAR-10 and Im-ageNet datasets. Moreover, we can successfully train plain CNNs to match the performance of the residual counterparts.Besides, we propose new principles for designing network structure from the insights evoked by orthonormality. Combined with residual structure, we achieve comparative performance on the ImageNet dataset.

ABD-Net: Attentive but Diverse Person Re-Identification

“…Recent studies also investigated "softer" orthogonality regularizations by enforcing the Gram matrix of each weight matrix to be close to an identity matrix, under Frobenius norm [38] or spectral norm [39]. We propose a novel spectral value difference orthogonality (SVDO) regularization that directly constrains the conditional number of the Gram matrix.…”

“…We propose a novel spectral value difference orthogonality (SVDO) regularization that directly constrains the conditional number of the Gram matrix. Also contrasting from [13,38] that apply orthogonality only to CNN weights, we enforce the new regularization on both hidden activations and weights.…”

“…However, computing SVD on highdimensional matrices is expensive, urging for the development of soft orthogonality regularizers. Many existing soft regularizers [38,41] restrict the Gram matrix of F to be close to an identity matrix under Frobenius norm that can avoid the SVD step while being differentiable. However, * For the weight tensor Wc ∈ R S×H×C×M in a convolutional layer, where S, H, C, M are filter's width and height, the number of input and output channels, we follow the convention of [38,39] to reshape Wc into a matrix form…”

Person Re-Identification in Aerial Imagery

“…Lots of research works have been focusing on learning compact and non-redundant feature descriptors which have many good merits [39], [40], [41], [20], [42], [21]. Chan et al [42] proposed a very simple deep learning framework called PCANet for image classification, which learned orthogonal projection to produce the filters.…”

“…Chan et al [42] proposed a very simple deep learning framework called PCANet for image classification, which learned orthogonal projection to produce the filters. Xie et al [41] utilized the regularization effect of orthogonalization to improve the classification accuracy. Sun et al [39] quantitatively analyzed the influence of features on person ReID accuracy and found that the associations between different features also impacted the results.…”