Generalization bounds for deep learning

Abstract

Generalization in deep learning has been the topic of much recent theoretical and empirical research. Here we introduce desiderata for techniques that predict generalization errors for deep learning models in supervised learning. Such predictions should 1) scale correctly with data complexity; 2) scale correctly with training set size; 3) capture differences between architectures; 4) capture differences between optimization algorithms; 5) be quantitatively not too far from the true error (in particular, be non-vacuous); 6) be efficiently computable; and 7) be rigorous. We focus on generalization error upper bounds, and introduce a categorisation of bounds depending on assumptions on the algorithm and data. We review a wide range of existing approaches, from classical VC dimension to recent PAC-Bayesian bounds, commenting on how well they perform against the desiderata. We next use a function-based picture to derive a marginal-likelihood PAC-Bayesian bound. This bound is, by one definition, optimal up to a multiplicative constant in the asymptotic limit of large training sets, as long as the learning curve follows a power law, which is typically found in practice for deep learning problems. Extensive empirical analysis demonstrates that our marginal-likelihood PAC-Bayes bound fulfills desiderata 1-3 and 5. The results for 6 and 7 are promising, but not yet fully conclusive, while only desideratum 4 is currently beyond the scope of our bound. Finally, we comment on why this function-based bound performs significantly better than current parameter-based PAC-Bayes bounds.

The prevalence and characteristics of metabolic syndrome according to different definitions in China: a nationwide cross-sectional study, 2012–2015

“…For example, the complexity of the hypothesis space bound increases as the number of parameters becomes larger and the stability-based bound grows with respect to the iterations of optimization. To handle the problems and further improve the generalization bound of GNNs, researchers can leverage recent advances in deep learning theory such as local Rademacher complexity [178], marginal-likelihood PAC-Bayes [179] and H-consistency [180]. Besides, it is observed that existing generalization bounds often heavily rely on the number of nodes and maximum node degree as the graph-related term in their final expressions, which is too coarse-grained to capture the complex graph structure information.…”

A New Measure of Model Redundancy for Compressed Convolutional Neural Networks

“…A comprehensive review by Valle-Pérez and Louis (2020) shows that, most types of generalization bounds are characterized by the norm of network weights or other related quantities such as the Lipschitz constant of the network; some examples include the margin-based bounds (Bartlett et al, 2017), the sensitivity-based bounds (Neyshabur et al, 2017), the NTK-based bounds (Cao and Gu, 2019) or the compression-based bounds (Li et al, 2020). They are model-agnostic in the sense that, only a generic functional form of the network is required and the weights are then implicitly regularized by some learning algorithms.…”

Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functions

“…Generalization in reinforcement learning Generalizing a model's predictions across a variety of unseen, high-dimensional inputs has been extensively studied in the static supervised learning setting [Bartlett, 1998, Triantafillou et al, 2019, Valle-Pérez and Louis, 2020, Liu et al, 2020b. Gener-…”