A study of transformation-invariances of deep belief networks

Abstract

In order to learn transformation-invariant features, several effective deep architectures like hierarchical feature learning and variant Deep Belief Networks (DBN) have been proposed. Considering the complexity of those variants, people are interested in whether DBN itself has transformation-invariances. First of all, we use original DBN to test original data. Almost same error rates will be achieved, if we change weights in the bottom interlayer according to transformations occurred in testing data. It implies that weights in the bottom interlayer can store the knowledge to handle transformations such as rotation, shifting, and scaling. Along with the continuous learning ability and good storage of DBN, we present our Weight-Transformed Training Algorithm (WTTA) without augmenting other layers, units or filters to original DBN. Based upon original training method, WTTA is aiming at transforming weights and is still unsupervised. For MNIST handwritten digits recognizing experiments, we adopted 784-100-100-100 DBN to compare the differences of recognizing ability in weights-transformed ranges. Most error rates generated by WTTA were below 25% while most rates generated by original training algorithm exceeded 25%. Then we also did an experiment on part of MIT-CBCL face database, with varying illumination, and the best testing accuracy can be achieved is 87.5%. Besides, similar results can be achieved by datasets covering all kinds of transformations, but WTTA only needs original training data and transform weights after each training loop. Consequently, we can mine inherent transformation-invariances of DBN by WTTA, and DBN itself can recognize transformed data at satisfying error rates without inserting other components.

Rotation-Invariant Restricted Boltzmann Machine Using Shared Gradient Filters

“…Current approaches do not address the problem of rotation-invariance directly, but use a predefined set of transformations to transform either the input images [19,21] or the learned filters [13,20]. We were inspired by these approaches to modify the RBM learning process, such that to learn invariant features without taking into account all possible transformations, which is demanding and may propagate noise due to pixel interpolations.…”

“…Although RBMs can achieve satisfactory results [4], their use in shallow networks (namely few layers) cannot accommodate complex variability occurring in the scene [20]. To this end, the Deep Belief Network (DBN) was proposed in [14], which is constituted by several stacked RBMs.…”

“…The marginal probability of RBM is extended with a Markov Random Field, including transformed versions of input images. In [20], an additional step of the backpropagation algorithm used to train DBN is introduced, where the weights are transformed and the entire network is trained again. In [3], the authors propose an RBM where input images are divided into non-overlapping blocks.…”

“…Then, patches are extracted on SIFT keypoints [18] and subsequently rotated and scaled accordingly. Despite their progress, the aforementioned methods share the following drawbacks: either they are limited to the set of transformations considered within the model, or they involve deep networks in the hope of learning better transformation invariant features [13,20,21], albeit increasing computational demand.…”

“…This information is used to select a particular weight matrix during the Gibbs sampling to compute gradients of parameters. The contribution given by the new update gradients is shared among the other weight matrices, rotating the filters accordingly [20] (cf. Figure 1).…”