Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, &amp; Gradient Flow Dynamics

Sahs, Justin; Pyle, Ryan; Damaraju, Aneel; Caro, Josue Ortega; Tavaslioglu, Onur; Lu, Andy; Patel, Ankit

doi:10.48550/arxiv.2008.01772

Cited by 7 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Independent of and concurrent with previous versions (Sahs et al, 2020a , b ) of this work, Williams et al ( 2019 ) has implicit regularization results in the kernel and adaptive regimes which parallel our results in this section rather closely. Despite the similarities, we take a significantly different approach.…”

Section: Theoretical Resultssupporting

confidence: 87%

Shallow Univariate ReLU Networks as Splines: Initialization, Loss Surface, Hessian, and Gradient Flow Dynamics

Sahs

Pyle

Damaraju

et al. 2022

Front. Artif. Intell.

Self Cite

View full text Add to dashboard Cite

Understanding the learning dynamics and inductive bias of neural networks (NNs) is hindered by the opacity of the relationship between NN parameters and the function represented. Partially, this is due to symmetries inherent within the NN parameterization, allowing multiple different parameter settings to result in an identical output function, resulting in both an unclear relationship and redundant degrees of freedom. The NN parameterization is invariant under two symmetries: permutation of the neurons and a continuous family of transformations of the scale of weight and bias parameters. We propose taking a quotient with respect to the second symmetry group and reparametrizing ReLU NNs as continuous piecewise linear splines. Using this spline lens, we study learning dynamics in shallow univariate ReLU NNs, finding unexpected insights and explanations for several perplexing phenomena. We develop a surprisingly simple and transparent view of the structure of the loss surface, including its critical and fixed points, Hessian, and Hessian spectrum. We also show that standard weight initializations yield very flat initial functions, and that this flatness, together with overparametrization and the initial weight scale, is responsible for the strength and type of implicit regularization, consistent with previous work. Our implicit regularization results are complementary to recent work, showing that initialization scale critically controls implicit regularization via a kernel-based argument. Overall, removing the weight scale symmetry enables us to prove these results more simply and enables us to prove new results and gain new insights while offering a far more transparent and intuitive picture. Looking forward, our quotiented spline-based approach will extend naturally to the multivariate and deep settings, and alongside the kernel-based view, we believe it will play a foundational role in efforts to understand neural networks. Videos of learning dynamics using a spline-based visualization are available at http://shorturl.at/tFWZ2.

show abstract

Section: Theoretical Resultssupporting

confidence: 87%

Shallow Univariate ReLU Networks as Splines: Initialization, Loss Surface, Hessian, and Gradient Flow Dynamics

Sahs

Pyle

Damaraju

et al. 2022

Front. Artif. Intell.

Self Cite

View full text Add to dashboard Cite

show abstract

“…While in practice it is more common that the weights in the hidden layer have a smaller variance (e.g. Glorot and Bengio [2010], He et al [2015]), our scaling prevents the breakpoints of the neurons in the trained network to move too much before we are able to decrease the training error sufficiently, and the impact of the magnitude by which we scale the hidden layer upon initialization on the dynamics of GF was studied in a similar univariate regression setting [Williams et al, 2019, Sahs et al, 2020. We now conclude the discussion of our main result with the following remarks on the setting studied in our paper.…”

Section: Resultsmentioning

confidence: 99%

On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias

Safran¹,

Vardi²,

Lee³

2022

Preprint

View full text Add to dashboard Cite

We study the dynamics and implicit bias of gradient flow (GF) on univariate ReLU neural networks with a single hidden layer in a binary classification setting. We show that when the labels are determined by the sign of a target network with r neurons, with high probability over the initialization of the network and the sampling of the dataset, GF converges in direction (suitably defined) to a network achieving perfect training accuracy and having at most O(r) linear regions, implying a generalization bound. Our result may already hold for mild over-parameterization, where the width is Õ(r) and independent of the sample size.

show abstract

“…It is interesting that even when allowing for more input arguments, the resultant learned nonlinearities favor low-order quadratic functions (Figure 8b-d). This could be explained by an implicit bias toward smooth functions [45,46] while still bending the input space to provide useful computations. Perhaps the learned nonlinearities are as random as possible while fulfilling these minimal conditions.…”

Section: Discussionmentioning

confidence: 99%

Two-Argument Activation Functions Learn Soft XOR Operations Like Cortical Neurons

et al. 2022

View full text Add to dashboard Cite

Neurons in the brain are complex machines with distinct functional compartments that interact nonlinearly. In contrast, neurons in artificial neural networks abstract away this complexity, typically down to a scalar activation function of a weighted sum of inputs. Here we emulate more biologically realistic neurons by learning canonical activation functions with two input arguments, analogous to basal and apical dendrites. We use a network-in-network architecture where each neuron is modeled as a multilayer perceptron with two inputs and a single output. This inner perceptron is shared by all units in the outer network. Remarkably, the resultant nonlinearities often produce soft XOR functions, consistent with recent experimental observations about interactions between inputs in human cortical neurons. When hyperparameters are optimized, networks with these nonlinearities learn faster and perform better than conventional ReLU nonlinearities with matched parameter counts, and they are more robust to natural and adversarial perturbations.

show abstract

Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamics

Cited by 7 publications

References 0 publications

Shallow Univariate ReLU Networks as Splines: Initialization, Loss Surface, Hessian, and Gradient Flow Dynamics

Shallow Univariate ReLU Networks as Splines: Initialization, Loss Surface, Hessian, and Gradient Flow Dynamics

On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias

Two-Argument Activation Functions Learn Soft XOR Operations Like Cortical Neurons

Contact Info

Product

Resources

About