Exact Solutions of a Deep Linear Network

Abstract

This work finds the exact solutions to a deep linear network with weight decay and stochastic neurons, a fundamental model for understanding the landscape of neural networks. Our result implies that weight decay strongly interacts with the model architecture and can create bad minima in a network with more than 1 hidden layer, qualitatively different for a network with only 1 hidden layer. As an application, we also analyze stochastic nets and show that their prediction variance vanishes to zero as the stochasticity, the width, or the depth tends to infinity.

Exact Phase Transitions in Deep Learning

“…The existing initialization methods are predominantly data-dependent. However, our result (also see [24]) suggests that the size of the trivial minimum is data-dependent, and our result thus highlights the importance of designing data-dependent initialization methods in deep learning.…”

“…where x is the input data, y the label, U and W (i) the model parameters, D the network depth, the noise in the hidden layer (e.g., dropout), d 0 the width of the model, and γ the weight decay strength. We build on the recent results established in [24]. Let b ∶= U d 0 .…”

“…Ref. [24] shows that all the global minima of Eq. ( 5) must take the form U = f (b) and W i = f i (b), where f and f i are explicit functions of the hyperparameters.…”

“…Ref. [24] further shows that there are two regimes of learning, where, for some range of γ, the global minimum is uniquely given by b = 0, and for some other range of γ, some b > 0 gives the global minimum. When b = 0, the model outputs a constant 0, and so this regime is called the "trivial regime," and the regime where b = 0 is not the global minimum is called the "feature learning regime."…”

“…By Theorem 1 of Ref. [24], the phase transition, if exists, must occur precisely at γ = E[xy] . To prove that γ = E[xy] has a second-order phase transition, we must check both its first derivative and second derivative.…”