Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization

Abstract

While stochastic gradient descent (SGD) and variants have been surprisingly successful for training deep nets, several aspects of the optimization dynamics and generalization are still not well understood.In this paper, we present new empirical observations and theoretical results on both the optimization dynamics and generalization behavior of SGD for deep nets based on the Hessian of the training loss and associated quantities. We consider three specific research questions:(1) what is the relationship between the Hessian of the loss and the second moment of stochastic gradients (SGs)? (2) how can we characterize the stochastic optimization dynamics of SGD with fixed and adaptive step sizes and diagonal pre-conditioning based on the first and second moments of SGs? and (3) how can we characterize a scale-invariant generalization bound of deep nets based on the Hessian of the loss, which by itself is not scale invariant? We shed light on these three questions using theoretical results supported by extensive empirical observations,with experiments on synthetic data, MNIST, and CIFAR-10, with different batch sizes, and with different difficulty levels by synthetically adding random labels.Our experiments explore the fully connected feed-forward network with Relu activations. We evaluate SGD dynamics on both synthetic datasets and some commonly used real datasets, viz., the MNIST database of handwritten digits [38] and the CIFAR-10 dataset [33]. The synthetic datasets, which are inspired by recent work in [64,80], consist of equal number of samples drawn from k isotropic Gaussians with different means, each corresponding to one class. We refer to these datasets as Gauss-k. We also consider variants of these datasets where a fixed fraction of points have been assigned random labels [80] without changing the features. Details of our experimental setup are discussed in the Appendix. Experiments on both Gauss-10 and Gauss-2 datasets are repeated 10,000 times for each setting to get a full distributional characterization of the loss stochastic process and associated quantities, including full eigen-spectrum of the Hessian of the loss and the second moment. In the main paper, we present results based on Gauss-10 and some on MNIST and CIFAR-10. Additional results on all datasets and all proofs are discussed in the Appendix.

Empirically explaining SGD from a line search perspective

“…Finally, Figure 6 suggests that exact line searches on the mini-batch loss perform poorly. Supporting results are obtained for SGD with momentum, for ResNet-18 and for Mo-bileNetV2 see Appendix Figures 9,15,16,17,18. However, in the case of SGD with momentum the line search is constantly not as exact.…”

“…SGD trajectories: Similar to this work [29] analyzes the loss along SGD trajectories, but with less focus on line searches and the exact shape of the full-batch loss. [12] and [16] consider second order information along SGD trajectories. Where [12] investigates the spectral norm of the Hessian (highest curvature) along the SGD trajectory and shows, inter alia, that it initially visits increasingly sharp regions.…”

“…Where [12] investigates the spectral norm of the Hessian (highest curvature) along the SGD trajectory and shows, inter alia, that it initially visits increasingly sharp regions. [16] investigates the dynamics and generalization of SGD based on the Hessian of the loss. They show, among other things, that the primary subspace of the second momentum of stochastic gradients overlaps substantially with that of the Hessian.…”

Large Scale Private Learning via Low-rank Reparametrization

“…The phenomenon that the gradients of deep models live on a very low dimensional manifold has been widely observed (Gur-Ari et al, 2018;Vogels et al, 2019;Gooneratne et al, 2020;Li et al, 2020;Martin & Mahoney, 2018;Li et al, 2018). People have also used this fact to compress the gradient with low-rank approximation in the distributed optimization scenario (Yurtsever et al, 2017;Wang et al, 2018b;Karimireddy et al, 2019;Vogels et al, 2019).…”

Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning

“…The dimensional barrier is attributed to the fact that the added noise is isotropic while the gradients live on a very low dimensional manifold, which has been observed in (Gur-Ari et al, 2018;Vogels et al, 2019;Gooneratne et al, 2020;Li et al, 2020) and is also verified in Figure 2 for the gradients of a 20-layer ResNet (He et al, 2016). Hence to limit the noise energy, it is natural to think "Can we reduce the dimension of gradients first and then add the isotropic noise onto a low-dimensional gradient embedding?…”