Backward Feature Correction: How Deep Learning Performs Deep Learning

Abstract

How does a 110-layer ResNet learn a high-complexity classifier using relatively few training examples and short training time? We present a theory towards explaining this in terms of hierarchical learning. We refer hierarchical learning as the learner learns to represent a complicated target function by decomposing it into a sequence of simpler functions to reduce sample and time complexity. This paper formally analyzes how multi-layer neural networks can perform such hierarchical learning efficiently and automatically simply by applying stochastic gradient descent (SGD) to the training objective.On the conceptual side, we present, to the best of our knowledge, the first theory result indicating how very deep neural networks can still be sample and time efficient on certain hierarchical learning tasks, when no known non-hierarchical algorithms (such as kernel method, linear regression over feature mappings, tensor decomposition, sparse coding, and their simple combinations) are efficient. We establish a new principle called "backward feature correction", which we believe is the key to understand the hierarchical learning in multi-layer neural networks.On the technical side, we show for regression and even for binary classification, for every input dimension d > 0, there is a concept class consisting of degree ω(1) multi-variate polynomials so that, using ω(1)-layer neural networks as learners, SGD can learn any target function from this class in poly(d) time using poly(d) samples to any 1 poly(d) regression or classification error, through learning to represent it as a composition of ω(1) layers of quadratic functions. In contrast, we present lower bounds stating that several non-hierarchical learners, including any kernel methods, neural tangent kernels, must suffer from super-polynomial d ω(1) sample or time complexity to learn functions in this concept class even to any d −0.01 error.

Optimization-Based Separations for Neural Networks

“…Additionally, the authors also demonstrate experimentally that depth 3 networks outperform depth 2 networks for learning ball indicators, even if the shallow network has many more tweakable weights. While we build on a similar proof strategy as in Safran and Shamir [25], our work differs from theirs in that they show that for a ball indicator function of any radius, there exists a distribution under which it is hard to approximate using depth 2 networks, whereas our work shows that for a specific distribution there exists a ball indicator function with radius in the interval [1,2] which is hard to approximate using a depth 2 network. Moreover, our main result rigorously proves the learnability of the target function used in their experiment.…”

“…• We show that there exist a sequence {D d } ∞ d=2 of d-dimensional distributions and a sequence of constants {λ d } ∞ d=2 , where λ d ∈ [1,2] for all d, such that no neural network of depth 2 and width less than Ω (exp(Ω(d))) can approximate ball indicator functions with radii {λ d } ∞ d=2 to accuracy better than Ω(d −4 ) (Thm. 2.2).…”

“…• We prove that under certain assumptions on the distribution of the data (Assumption 3.1), a neural network with two layers of non-linear activations (see Eq. ( 2) for the formal architecture) can efficiently learn data labeled by a ball indicator function with radius in [1,2], and attain arbitrarily small population loss by using GD with a standard initialization (Assumption 3.2), where the hidden layer is held fixed throughout training (Thm. 3.3).…”

“…In Allen-Zhu et al [2], the authors prove that a certain class of depth 3 target functions can be improperly learned using depth 3 networks with a different architecture. Allen-Zhu and Li [1] show that deep networks with quadratic activations can heirarchically learn certain concept classes. However, the learning algorithm used in Goel and Klivans [13] is different than GD, and all three papers crucially rely on the input having unit norm or some other form of boundedness, whereas our work accommodates for certain inputs whose norm distribution does not have finite moments.…”

“…where σ is some non-linear activation function which satisfies Assumptions 1 and 2 in Eldan and Shamir [10]. 1 As the stronger class with superior approximation capabilities, we consider the following 'depth 3' neural networks with a simplified structure.…”