Rademacher complexity and spin glasses: A link between the replica and statistical theories of learning

Abstract

Statistical learning theory provides bounds of the generalization gap, using in particular the Vapnik-Chervonenkis dimension and the Rademacher complexity. An alternative approach, mainly studied in the statistical physics literature, is the study of generalization in simple synthetic-data models. Here we discuss the connections between these approaches and focus on the link between the Rademacher complexity in statistical learning and the theories of generalization for typical-case synthetic models from statistical physics, involving quantities known as Gardner capacity and ground state energy. We show that in these models the Rademacher complexity is closely related to the ground state energy computed by replica theories. Using this connection, one may reinterpret many results of the literature as rigorous Rademacher bounds in a variety of models in the high-dimensional statistics limit. Somewhat surprisingly, we also show that statistical learning theory provides predictions for the behavior of the ground-state energies in some full replica symmetry breaking models.

Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization

“…This is much worse that what we observe in practice, where we reach the Bayes rate e g = Θ(α −1 ). Tighter bounds can be obtained using the Rademacher complexity, and this was studied recently (using the aforementioned replica method) in [49] for the very same problem. We reproduced their results and plotted the Rademacher complexity generalization bound in Fig.…”

Statistical learning theory of structured data

“…Indeed, one of the main pursuits of contemporary SLT is to provide better results on the generalization error, going beyond distribution independent bounds. Several strategies have been proposed, advocating the importance of considering datadependent hypothesis classes [25] and data-dependent measures of complexity (such as the Rademacher complexity [36], which was recently connected to the statistical mechanics of disordered systems [37]), also in relation to the original concept of VC entropy itself [38].…”

“…This approximation has negligible effects in the large-n limit [31]. Equation (37) fixes the recurrence relation satisfied by the generating function g n (z):…”