No abstract

It is common practice in deep learning to represent a measurement of the world on a discrete grid, e.g. a 2D grid of pixels. However, the underlying signal represented by these measurements is often continuous, e.g. the scene depicted in an image. A powerful continuous alternative is then to represent these measurements using an implicit neural representation, a neural function trained to output the appropriate measurement value for any input spatial location. In this paper, we take this idea to its next level: what would it take to perform deep learning on these functions instead, treating them as data? In this context we refer to the data as functa, and propose a framework for deep learning on functa. This view presents a number of challenges around efficient conversion from data to functa, compact representation of functa, and effectively solving downstream tasks on functa. We outline a recipe to overcome these challenges and apply it to a wide range of data modalities including images, 3D shapes, neural radiance fields (NeRF) and data on manifolds. We demonstrate that this approach has various compelling properties across data modalities, in particular on the canonical tasks of generative modeling, data imputation, novel view synthesis and classification.

BACKGROUND ON NEURAL FIELDS AND FUNCTANeural fields are functions f θ mapping coordinates x (e.g. pixel locations) to features f (e.g. RGB values) parameterized by θ. They are fit to a particular signal by minimizing the reconstruction loss between the output f and the target signal value at all available coordinates x. For example, an image would correspond to a single f θ , usually an MLP with sinusoidal activation functions, i.e., a SIREN (Sitzmann et al., 2020). Neural fields are very general and can represent a wide range of the aforementioned modalities.Functa (Dupont et al., 2022a) is a framework that treats neural fields as data points, elements of a functaset, on which we can perform deep learning tasks. See App. A for a more thorough introduction. The motivations for doing so include: (1) better scaling of data dimensionality compared to array representations, leading to more memory/compute efficient training of neural networks for downstream tasks; (2) moving away from a fixed resolution; (3) allowing for unified frameworks and architectures across multiple data modalities; (4) dealing with signals that are inherently difficult to discretize, such as fields on non-Euclidean manifolds, e.g., climate data. Dupont et al. (2022a) show promising results on relatively small scale datasets with limited complexity. In this work we ask the question: does the approach scale to more complex / larger datasets? If not, how can we make it scale?3 LIMITATIONS OF NAIVELY SCALING UP FUNCTA Following implementation details by Dupont et al. (2022a), we were able to reproduce reported results on CelebA-HQ (64 × 64) and subsequently applied functa to . We discovered that, while the meta-learned functa latent representations could faithfully reconstruct the data, on downstream tasks they significantly underperformed pixel-based classification and generation.More specifically, we were able to fit each CIFAR-10 image successfully with meta-learning, that is, we could reconstruct test set images to very high fidelity (38.1dB peak signal to noise ratio (PSNR))