Limitation of characterizing implicit regularization by data-independent functions

Abstract

In recent years, understanding the implicit regularization of neural networks (NNs) has become a central task of deep learning theory. However, implicit regularization is in itself not completely defined and well understood. In this work, we make an attempt to mathematically define and study the implicit regularization. Importantly, we explore the limitation of a common approach of characterizing the implicit regularization by dataindependent functions. We propose two dynamical mechanisms, i.e., Two-point and One-point Overlapping mechanisms, based on which we provide two recipes for producing classes of onehidden-neuron NNs that provably cannot be fully characterized by a type of or all data-independent functions. Our results signify the profound datadependency of implicit regularization in general, inspiring us to study in detail the data-dependency of NN implicit regularization in the future.

The Weights Reset Technique for Deep Neural Networks Implicit Regularization

“…In [11], the authors argue that many previous works have focused on characterizing implicit regularization using functions that are independent of the training data, such as the norm of the weight matrix or the singular values of the data matrix. While these functions can provide useful insights into the behavior of deep neural networks, they are limited in their ability to capture the full complexity of the optimization landscape.…”