Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization

De-randomized PAC-Bayes Margin Bounds: Applications to Non-convex and Non-smooth Predictors

“…The diagonal elements have an interesting empirical behavior (Section 5): a small number of diagonal elements have relatively high values and most have quite small values. Such behavior aligns well with recent results on the eigen-spectrum of the Hessian [Li et al, 2019, Sagun et al, 2016. Further, all the diagonal elements decrease as more samples are used for training.…”

“…One concern in using the Hessian H θ † l,φ is its scale dependence [Dinh et al, 2017], but we prove that our bound is scale invariant (Appendix B). While H θ † l,φ could have been constructed from the Hessian eigen-values [Li et al, 2019, Sagun et al, 2016, the resulting bound would have been scale dependent and hence undesirable. Further, the diagonals of the Hessian are much easier to numerically compute compared to the eigen-values.…”

“…However, for deep nets with positively homogeneous activations, the measure of flatness/sharpness can be arbitrarily changed through rescaling ("α-scale transformation" Dinh et al [2017]) without changing the generalization performance, rendering the notion of 'flatness' meaningless. Thus, there are some works Li et al [2019] exploring scale-invariant generalization bound for the deep network. In the perspective of explaining generalization by Rademacher complexity, Bartlett et al [2017], Neyshabur et al [2015], Golowich et al [2018] derive bound on the Rademacher complexity of ReLU nets, which depends on the product of the spectral norms of layers.…”

“…In particular, we propose a Gaussian posterior with a diagonal covariance matrix where parameters corresponding to flat directions, i.e., loss is insensitive to changes in these parameters, have the same variance as the prior whereas parameters corresponding to sharp curvature directions, i.e., loss is sensitive to changes in these parameters, have variance which is inversely proportional to the corresponding Hessian diagonal elements and hence small. In spite of the dependency on the Hessian diagonal elements, which can be changed based on re-parameterization without changing the function [Smith andLe, 2018, Dinh et al, 2017], the bound itself is scale invariant since KL-divergence is invariant to such re-parameterizations Kleeman [2011], Li et al [2019]. The resulting bound depends on a trade-off between effective curvature and distance from the initialization of the learned deep net parameters, where the trade-off gets determined by the marginal prior variance.…”