De-randomized PAC-Bayes Margin Bounds: Applications to Non-convex and Non-smooth Predictors

Abstract

In spite of several notable efforts, explaining the generalization of deterministic deep nets, e.g., ReLUnets, has remained challenging. Existing approaches usually need to bound the Lipschitz constant of such deep nets but such bounds have been shown to increase substantially with the number of training samples yielding vacuous generalization bounds [Nagarajan and Kolter, 2019a]. In this paper, we present new de-randomized PAC-Bayes margin bounds for deterministic non-convex and non-smooth predictors, e.g., ReLU-nets. The bounds depend on a trade-off between the L2-norm of the weights and the effective curvature ('flatness') of the predictor, avoids any dependency on the Lipschitz constant, and yield meaningful (decreasing) bounds with increase in training set size. Our analysis first develops a de-randomization argument for non-convex but smooth predictors, e.g., linear deep networks (LDNs). We then consider non-smooth predictors which for any given input realize as a smooth predictor, e.g., ReLU-nets become some LDN for a given input, but the realized smooth predictor can be different for different inputs. For such non-smooth predictors, we introduce a new PAC-Bayes analysis that maintains distributions over the structure as well as parameters of smooth predictors, e.g., LDNs corresponding to ReLU-nets, which after de-randomization yields a bound for the deterministic non-smooth predictor. We present empirical results to illustrate the efficacy of our bounds over changing training set size and randomness in labels.

Generalization bounds for deep learning

“…Recently, Banerjee et al (2020) have developed novel deterministic bounds based on a de-randomization of a PAC-Bayes bound. Their bound is also based on the flatness of the minimum found after training (measured by Hessian eigenvalues), and also takes into account the distance moved in parameter space.…”

“…Neyshabur et al (2017);Banerjee et al (2020) provided some evidence that their PAC-Bayesian bounds correlates with true error when varying the data complexity. The authors are not aware of similar results for the other measures.• D.2 Neyshabur et al (2017); Banerjee et al (2020); Dziugaite et al (2020) have shown evidence that different sensitivity-based PAC-Bayesian bounds correlate with true error when varying the training set size.…”