A Structured Dictionary Perspective on Implicit Neural Representations

Abstract

Propelled by new designs that permit to circumvent the spectral bias, implicit neural representations (INRs) have recently emerged as a promising alternative to classical discretized representations of signals. Nevertheless, despite their practical success, we still lack a proper theoretical characterization of how INRs represent signals. In this work, we aim to fill this gap, and we propose a novel unified perspective to theoretically analyse INRs. Leveraging results from harmonic analysis and deep learning theory, we show that most INR families are analogous to structured signal dictionaries whose atoms are integer harmonics of the set of initial mapping frequencies. This structure allows INRs to express signals with an exponentially increasing frequency support using a number of parameters that only grows linearly with depth. Afterwards, we explore the inductive bias of INRs exploiting recent results about the empirical neural tangent kernel (NTK). Specifically, we show that the eigenfunctions of the NTK can be seen as dictionary atoms whose inner product with the target signal determines the final performance of their reconstruction. In this regard, we reveal that meta-learning the initialization has a reshaping effect of the NTK analogous to dictionary learning, building dictionary atoms as a combination of the examples seen during meta-training. Our results permit to design and tune novel INR architectures, but can also be of interest for the wider deep learning theory community.

Sobolev Training for Implicit Neural Representations with Approximated Image Derivatives

“…Network Architecture For network architecture, we implement a fully-connected MLP with 4 activated linear layers with 256 hidden units each and 1 output linear layer. For the ReLU-based implementation, we follow the conclusion in [43,36] and apply positional encoding (P.E.) to the inputs (additional 20 channels).…”

“…The angular velocity of the sinusoidal function and weight initialization method are both the same as image regression task. Taking the conclusion in [36,43] into consideration, we do two parallel experiments with and without positional encoding for all activation functions except for sine.…”