Overview frequency principle/spectral bias in deep learning

Xu, Zhi‐Qin John; Zhang, Yaoyu; Luo, T.

doi:10.48550/arxiv.2201.07395

Cited by 12 publications

(17 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the condensation regime, a large network with condensation is effectively a network of only a few effective neurons, leading to an output function with low complexity (Bartlett and Mendelson, 2002), thus, may provide a possible explanation for good generalization performance of large NNs (Breiman, 1995;. The condensation suggests a bias towards simple function, which is consistent with the frequency principle that NNs tend to learn data by low-frequency function (Xu et al, 2019;Rahaman et al, 2019;Xu et al, , 2022. NNs have significant feature learning in the condensation regimes.…”

Section: Related Workmentioning

confidence: 78%

Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width

Zhou¹,

Zhou²,

Jin³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Substantial work indicates that the dynamics of neural networks (NNs) is closely related to their initialization of parameters. Inspired by the phase diagram for two-layer ReLU NNs with infinite width , we make a step towards drawing a phase diagram for three-layer ReLU NNs with infinite width. First, we derive a normalized gradient flow for three-layer ReLU NNs and obtain two key independent quantities to distinguish different dynamical regimes for common initialization methods. With carefully designed experiments and a large computation cost, for both synthetic datasets and real datasets, we find that the dynamics of each layer also could be divided into a linear regime and a condensed regime, separated by a critical regime. The criteria is the relative change of input weights (the input weight of a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term) as the width approaches infinity during the training, which tends to 0, +∞ and O(1), respectively. In addition, we also demonstrate that different layers can lie in different dynamical regimes in a training process within a deep NN. In the condensed regime, we also observe the condensation of weights in isolated orientations with low complexity. Through experiments under three-layer condition, our phase diagram suggests a complicated dynamical regimes consisting of three possible regimes, together with their mixture, for deep NNs and provides a guidance for studying deep NNs in different initialization regimes, which reveals the possibility of completely different dynamics emerging within a deep NN for its different layers.

show abstract

Section: Related Workmentioning

confidence: 78%

Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width

Zhou¹,

Zhou²,

Jin³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Frequency loss. As studied in previous works [40,47,56] on the learning behavior from frequency domain, spectral bias of the deep neural networks is often inclined to low frequency functions. Besides, according to F-Principle [57], the fitting priority of a network to certain frequencies is various throughout the training, often in the low-to-high pattern.…”

Section: Pre-training Strategymentioning

confidence: 99%

The Devil is in the Frequency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Training

Líu¹,

Jiang²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

The self-supervised Masked Image Modeling (MIM) schema, following "mask-and-reconstruct" pipeline of recovering contents from masked image, has recently captured the increasing interest in the community, owing to the excellent ability of learning visual representation from unlabeled data. Aiming at learning representations with high semantics abstracted, a group of works attempts to reconstruct non-semantic pixels with large-ratio masking strategy, which may suffer from "over-smoothing" problem, while others directly infuse semantics into targets in offline way requiring extra data. Different from them, we shift the perspective to the Fourier domain which naturally has global perspective and present a new Masked Image Modeling (MIM), termed Geminated Gestalt Autoencoder (Ge 2 -AE) for visual pre-training. Specifically, we equip our model with geminated decoders in charge of reconstructing image contents from both pixel and frequency space, where each other serves as not only the complementation but also the reciprocal constraints. Through this way, more robust representations can be learned in the pre-trained encoders, of which the effectiveness is confirmed by the juxtaposing experimental results on downstream recognition tasks. We also conduct several quantitative and qualitative experiments to investigate the learning behavior of our method. To our best knowledge, this is the first MIM work to solve the visual pre-training through the lens of frequency domain.

show abstract

“…For general non-linear NNs, empirical studies suggest that NNs have an implicitly regularization towards lowcomplexity function during the training (Arpit et al, 2017;Kalimeris et al, 2019;Goldt et al, 2020;Jin et al, 2020). For example, the frequency principle (Xu et al, 2019;Rahaman et al, 2019;Zhang et al, 2021;Xu et al, 2022) quantifies the implicit regularization of "simple solution" by showing that NNs learn the data from low to high frequency, i.e., implicit low-frequency regularization. as NNs in the linear regime studied in Luo et al (2021) and NTK regime studied in Jacot et al (2018), the low-frequency regularization of non-linear NNs can be exactly formulated by an data-independent function (Zhang et al, 2021;Luo et al, 2020) which explicitly shows that NNs pick a lowfrequency function from multiple solutions.…”

Section: Related Workmentioning

confidence: 99%

Limitation of characterizing implicit regularization by data-independent functions

Leyang¹,

Xu²,

Luo³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In recent years, understanding the implicit regularization of neural networks (NNs) has become a central task of deep learning theory. However, implicit regularization is in itself not completely defined and well understood. In this work, we make an attempt to mathematically define and study the implicit regularization. Importantly, we explore the limitation of a common approach of characterizing the implicit regularization by dataindependent functions. We propose two dynamical mechanisms, i.e., Two-point and One-point Overlapping mechanisms, based on which we provide two recipes for producing classes of onehidden-neuron NNs that provably cannot be fully characterized by a type of or all data-independent functions. Our results signify the profound datadependency of implicit regularization in general, inspiring us to study in detail the data-dependency of NN implicit regularization in the future.

show abstract

Overview frequency principle/spectral bias in deep learning

Cited by 12 publications

References 52 publications

Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width

Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width

The Devil is in the Frequency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Training

Limitation of characterizing implicit regularization by data-independent functions

Contact Info

Product

Resources

About