The Spectral Bias of Polynomial Neural Networks

Moulik, Choraria,; Dadi, Leello Tadesse; Chrysos, Grigorios; Mairal, Julien; Cevher, Volkan

doi:10.48550/arxiv.2202.13473

Cited by 2 publications

(3 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, [16] reveal how PNNs' architecture relates to polynomial factorization. Kileel et al [33] and Choraria et al [11] studies the expressive power of PNNs. Our work establish the connection between PNNs and many neural fields such as MFN [22] and BACON [36].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Polynomial Neural Fields for Subband Decomposition and Manipulation

Yang¹,

Benaim²,

Jampani³

et al. 2023

Preprint

View full text Add to dashboard Cite

Neural fields have emerged as a new paradigm for representing signals, thanks to their ability to do it compactly while being easy to optimize. In most applications, however, neural fields are treated like black boxes, which precludes many signal manipulation tasks. In this paper, we propose a new class of neural fields called polynomial neural fields (PNFs). The key advantage of a PNF is that it can represent a signal as a composition of a number of manipulable and interpretable components without losing the merits of neural fields representation. We develop a general theoretical framework to analyze and design PNFs. We use this framework to design Fourier PNFs, which match state-of-the-art performance in signal representation tasks that use neural fields. In addition, we empirically demonstrate that Fourier PNFs enable signal manipulation applications such as texture transfer and scale-space interpolation. Code is available at https://github.com/stevenygd/PNF. * Equal contribution. Part of this work was done while Guandao was a student researcher at Google. 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

show abstract

Section: Related Workmentioning

confidence: 99%

“…Doing the same thing to IPE results in worse performance as shown in RIPE. 11. 28.16 29.66 PNF 36.45 27.07 29.74 24.6 95.98 86.30 83.33 72.66 RIPE-sup 50.11 11.91 26.64 10.69 99.62 15.51 73.24 11.27 IPE-sup 34.76 26.8 30.00 18.41 90.97 84.81 89.65 65.7…”

mentioning

confidence: 99%

Polynomial Neural Fields for Subband Decomposition and Manipulation

Yang¹,

Benaim²,

Jampani³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…To improve the limitations of DNNs in learning high frequency components, empirical works use Fourier features explicitly as part of the input [43]. A parametrization of polynomial DNNs has also been shown to speed up the learning of higher frequency components in two-layer networks [44]. Different notions of spectral priors have also been empirically investigated in graph neural networks [22] and elsewhere [45].…”

Section: Related Workmentioning

confidence: 99%

Spectral Regularization Allows Data-frugal Learning over Combinatorial Spaces

Aghazadeh¹,

Rajaraman²,

Tu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Data-driven machine learning models are being increasingly employed in several important inference problems in biology, chemistry, and physics which require learning over combinatorial spaces. Recent empirical evidence (see, e.g., [1,2,3]) suggests that regularizing the spectral representation of such models improves their generalization power when labeled data is scarce. However, despite these empirical studies, the theoretical underpinning of when and how spectral regularization enables improved generalization is poorly understood. In this paper, we focus on learning pseudo-Boolean functions and demonstrate that regularizing the empirical mean squared error by the L 1 norm of the spectral transform of the learned function reshapes the loss landscape and allows for data-frugal learning, under a restricted secant condition on the learner's empirical error measured against the ground truth function. Under a weaker quadratic growth condition, we show that stationary points which also approximately interpolate the training data points achieve statistically optimal generalization performance. Complementing our theory, we empirically demonstrate that running gradient descent on the regularized loss results in a better generalization performance compared to baseline algorithms in several data-scarce real-world problems.

show abstract

The Spectral Bias of Polynomial Neural Networks

Cited by 2 publications

References 24 publications

Polynomial Neural Fields for Subband Decomposition and Manipulation

Polynomial Neural Fields for Subband Decomposition and Manipulation

Spectral Regularization Allows Data-frugal Learning over Combinatorial Spaces

Contact Info

Product

Resources

About