From Optimization Dynamics to Generalization Bounds via Łojasiewicz Gradient Inequality

Abstract

Optimization and generalization are two essential aspects of machine learning. In this paper, we propose a framework to connect optimization with generalization by analyzing the generalization error based on the length of optimization trajectory under the gradient flow algorithm after convergence. Through our approach, we show that, with a proper initialization, gradient flow converges following a short path with an explicit length estimate. Such an estimate induces a length-based generalization bound, showing that short optimization paths after convergence are associated with good generalization, which also matches our numerical results. Our framework can be applied to broad settings. For example, we use it to obtain generalization estimates on three distinct machine learning models: underdetermined p linear regression, kernel regression, and overparameterized two-layer ReLU neural networks.

Generalization Error Bounds for Deep Neural Networks Trained by SGD

“…Lastly, we particularly mention the work [25]. This work is related to ours since we both consider the connection between optimization and generalization by path estimate.…”

“…Chaining and conditional MI are also explored to derive more accurate bounds [3,37]. Besides, other techniques and approaches used to bound generalization error include model compression [2], margin theory [21], path length estimate [25] and linear stability of optimization algorithms [28].…”

“…This work is related to ours since we both consider the connection between optimization and generalization by path estimate. In [25], the authors derive generalization bounds for the Gradient Flow (GF) on linear and nearly linear models. By comparison, we analyze GF, GD and SGD for non-linear deep neural networks (MLPs and CNNs).…”