A Deep Conditioning Treatment of Neural Networks

Abstract

We study the role of depth in training randomly initialized overparameterized neural networks. We give the first general result showing that depth improves trainability of neural networks by improving the conditioning of certain kernel matrices of the input data. This result holds for arbitrary non-linear activation functions, and we provide a characterization of the improvement in conditioning as a function of the degree of non-linearity and the depth of the network. We provide versions of the result that hold for training just the top layer of the neural network, as well as for training all layers, via the neural tangent kernel. As applications of these general results, we provide a generalization of the results of Das et al. (2019) showing that learnability of deep random neural networks with arbitrary non-linear activations (under mild assumptions) degrades exponentially with depth. Additionally, we show how benign overfitting can occur in deep neural networks via the results of Bartlett et al. (2019b).1. Extending our analysis to layers of different sizes and outputs of length greater than 1 poses no mathematical difficulty and is omitted for the sake of clarity of notation.

Hardness of Learning Neural Networks with Natural Weights

“…In Das et al [2019], it is shown that deep random neural networks (of depth ω(log(n))) with the sign activation function, are hard to learn in the statistical query (SQ) model. This result was recently extended by Agarwal et al [2020] to other activation functions, including the ReLU function. While their results hold for networks of depth ω(log(n)) and for SQ algorithms, our results hold for depth-2 networks and for all algorithms.…”

From Local Pseudorandom Generators to Hardness of Learning

“…Due to the empirical success of neural networks, there has been much effort to understand under what assumptions neural networks may be learned efficiently. This effort includes making assumptions on the input distribution [Li and Yuan, 2017, Brutzkus and Globerson, 2017, Du et al, 2017a,b, Du and Goel, 2018, the network's weights [Arora et al, 2014, Das et al, 2019, Agarwal et al, 2020, Goel and Klivans, 2017, or both [Janzamin et al, 2015, Tian, 2017, Bakshi et al, 2019. Hence, distribution-specific learning of neural networks is a central problem.…”