Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect

Wang, Yuqing; Chen, Minshuo; Zhao, Tuo; Tao, Molei

doi:10.48550/arxiv.2110.03677

Cited by 2 publications

(2 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Let A be the set of those x 0 such that starting from these x 0 the GD will arrive at 0 after some steps. Because in the current case 0 is an unstable stationary point, it is easy to show that A contains countable number of points and hence has zero Lebesgue measure [20]. We ignore the detailed proof here.…”

Section: A One-dimensional Analysismentioning

confidence: 93%

Beyond the Quadratic Approximation: The Multiscale Structure of Neural Network Loss Landscapes

Ma¹,

Kunin²,

Wu³

et al. 2022

JML

View full text Add to dashboard Cite

A quadratic approximation of neural network loss landscapes has been extensively used to study the optimization process of these networks. Though, it usually holds in a very small neighborhood of the minimum, it cannot explain many phenomena observed during the optimization process. In this work, we study the structure of neural network loss functions and its implication on optimization in a region beyond the reach of a good quadratic approximation. Numerically, we observe that neural network loss functions possesses a multiscale structure, manifested in two ways: (1) in a neighborhood of minima, the loss mixes a continuum of scales and grows subquadratically, and (2) in a larger region, the loss shows several separate scales clearly. Using the subquadratic growth, we are able to explain the Edge of Stability phenomenon [1, 2] observed for the gradient descent (GD) method. Using the separate scales, we explain the working mechanism of learning rate decay by simple examples. Finally, we study the origin of the multiscale structure and propose that the non-convexity of the models and the non-uniformity of training data is one of the causes. By constructing a two-layer neural network problem we show that training data with different magnitudes give rise to different scales of the loss function, producing subquadratic growth and multiple separate scales.

show abstract

Section: A One-dimensional Analysismentioning

confidence: 93%

Beyond the Quadratic Approximation: The Multiscale Structure of Neural Network Loss Landscapes

Ma¹,

Kunin²,

Wu³

et al. 2022

JML

View full text Add to dashboard Cite

show abstract

“…We let A contain those x 0 that starting from these x 0 the GD will arrive at 0 after some steps. Because in the current case 0 is an unstable stationary point, it is easy to show that A contains countable number of points and hence has zero Lebesgue measure [18]. We ignore the detailed proof here.…”

Section: An 1-d Analysismentioning

confidence: 93%

The Multiscale Structure of Neural Network Loss Functions: The Effect on Optimization and Origin

Ma¹,

Kunin²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Local quadratic approximation has been extensively used to study the optimization of neural network loss functions around the minimum. Though, it usually holds in a very small neighborhood of the minimum, and cannot explain many phenomena observed during the optimization process. In this work, we study the structure of neural network loss functions and its implication on optimization in a region beyond the reach of good quadratic approximation. Numerically, we observe that neural network loss functions possesses a multiscale structure, manifested in two ways: (1) in a neighborhood of minima, the loss mixes a continuum of scales and grows subquadratically, and (2) in a larger region, the loss shows several separate scales clearly. Using the subquadratic growth, we are able to explain the Edge of Stability phenomenon [4] observed for gradient descent (GD) method. Using the separate scales, we explain the working mechanism of learning rate decay by simple examples. Finally, we study the origin of the multiscale structure and propose that the non-uniformity of training data is one of its cause. By constructing a two-layer neural network problem we show that training data with different magnitudes give rise to different scales of the loss function, producing subquadratic growth or multiple separate scales.

show abstract

Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect

Cited by 2 publications

References 34 publications

Beyond the Quadratic Approximation: The Multiscale Structure of Neural Network Loss Landscapes

Beyond the Quadratic Approximation: The Multiscale Structure of Neural Network Loss Landscapes

The Multiscale Structure of Neural Network Loss Functions: The Effect on Optimization and Origin

Contact Info

Product

Resources

About