Greedy Layerwise Learning Can Scale to ImageNet

Abstract

Shallow supervised 1-hidden layer neural networks have a number of favorable properties that make them easier to interpret, analyze, and optimize than their deep counterparts, but lack their representational power. Here we use 1-hidden layer learning problems to sequentially build deep networks layer by layer, which can inherit properties from shallow networks. Contrary to previous approaches using shallow networks, we focus on problems where deep learning is reported as critical for success. We thus study CNNs on image classification tasks using the large-scale ImageNet dataset and the CIFAR-10 dataset. Using a simple set of ideas for architecture and training we find that solving sequential 1-hidden-layer auxiliary problems lead to a CNN that exceeds AlexNet performance on ImageNet. Extending this training methodology to construct individual layers by solving 2-and-3-hidden layer auxiliary problems, we obtain an 11-layer network that exceeds several members of the VGG model family on ImageNet, and can train a VGG-11 model to the same accuracy as end-to-end learning. To our knowledge, this is the first competitive alternative to end-to-end training of CNNs that can scale to ImageNet. We illustrate several interesting properties of these models theoretically and conduct a range of experiments to study the properties this training induces on the intermediate representations.

Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks

“…[18] used the layer-wise method to train residual blocks in ResNet sequentially, then refined the network with the standard end-to-end training. [19] studied the progressive separability of layer-wise trained supervised neural networks and demonstrated Greedy layer-wise Learning (GLL) can scale to large-scale datasets like ImageNet. Other attempts at supervised layer-wise learning involve a synthetic gradient [20] and a layer-wise loss that combines local classifier and similarity matching loss [21].…”

“…Greedy and Randomized Layer-wise Learning. We adapted the supervised Greedy Layer-wise Learning (GLL) [19] method to self-supervised learning by training convolutional layers sequentially with auxiliary heads and self-supervised loss, as shown in Figure 1. The base encoders are trained layer by layer.…”

“…Such learning can be done sequentially bottom up, or randomly, updating a different random layer for each batch. The sequential form of learning has been studied in a number of papers [16,17,18,19,20,21]. However the sequential approach requires rigid timing of the updates of each layer, which would seem unlikely in the biological setting.…”

Front Contribution instead of Back Propagation

“…We survey three categories of BP literature-(i) better hardware implementation of BP [15,16,31,11,32,25], (ii) workarounds to approximate BP [33,7,10], and (iii) biologically inspired algorithms. Biologically inspired algorithms can further be segregated into four types: (i) Inspired from biological observations [29,7,26,17], these works try to approximate BP with the intention resolve its biological implausibility, (ii) Propagation of an alternative to error [19,21], (iii) Leveraging local errors, the power of single layer networks, and layer wise pre-training to approximate BP [24,23,3], (iv) Resolving the locking problem using decoupling [14,6,12,1,20] and its variants [27,8,22,4]. We were deeply motivated by (ii), (iii), and (iv) while coming up with the idea of 'front contributions'-specifically, propagating something other than error, the idea of a single layer network, and decoupling, collectively inspire 'front contributions'.…”

Memorization in Overparameterized Autoencoders

“…using gradient descent, where F denotes the network function space. Studying inductive bias in the context of autoencoders is relevant since (1) components of convolutional autoencoders are building blocks of many CNNs; (2) layerwise pre-training using autoencoders is a standard technique to initialize individual layers of CNNs to improve training [2,5,8]; and (3) autoencoder architectures are used in many image-to-image tasks such as image segmentation or impainting [25]. Furthermore, the inductive bias that we characterize in autoencoders may apply to more general architectures.…”