Input-level Inductive Biases for 3D Reconstruction

Abstract

Queries for image 1 Input image pair Multiple view geometry inductive biases flatten() Output depth for image 1 Input matrix Generalist Perception ModelFigure 1. Input-level inductive biases. We explore 3D reconstruction using a generalist perception model, the recent Perceiver IO [20] which ingests a matrix of unordered and flattened inputs (e.g. pixels). The model is interrogated using a query matrix and generates an output for every query -in this paper the outputs are depth values for all pixels of the input image pair. We incorporate inductive biases useful for multiple view geometry into this generalist model without having to touch its architecture, by instead encoding them directly as additional inputs.

Depth Field Networks for Generalizable Multi-view Scene Representation

“…For the task of optical flow, Perceiver IO feeds positionally encoded images through a Transformer [18], rather than using a cost volume for processing. IIB [58] adapts the Perceiver IO architecture to generalized stereo estimation, proposing a novel epipolar parameterization as an additional input-level inductive bias. Building upon this baseline, we propose a series of geometry-preserving 3D data augmentation techniques designed to promote the learning of a geometrically-consistent latent scene representation.…”

“…Our proposed DeFiNe architecture (Figure 2a) is designed with flexibility in mind, so data from different sources can be used as input and different output tasks can be estimated from the same latent space. Similar to Yifan et al [58], we use Perceiver IO [18] as our general-purpose Transformer backbone. During the encoding stage, our model takes RGB images from calibrated cameras, with known intrinsics and relative poses.…”

“…Data augmentation is a core component of deep learning pipelines [41] that improves model robustness by applying transformations to the training data consistent with the data distribution in order to introduce desired equivariant properties. In computer vision and depth estimation in particular, standard data augmentation techniques are usually constrained to the 2D space and include color jittering, flipping, rotation, cropping, and resizing [12,58]. Recent works have started looking into 3D augmentations [38] to improve robustness to errors in scene geometry in terms of camera localization (i.e., extrinsics) and parameters (i.e., intrinsics).…”

“…To test the benefits our proposed geometric 3D augmentation procedures over the IIB [58] baseline, we first evaluate DeFiNe on the task of stereo depth estimation.…”

“…Perceiver IO [17] extends this architecture to allow for arbitrary outputs, with results on optical flow estimation that outperform traditional cost-volume based methods. Similarly, the recent Input-level Inductive Bias (IIB) architecture [58] uses image features and camera information as input to Perceiver IO to directly regress stereo depth, outperforming baselines that use explicit geometric constraints. Building upon these works, we propose to learn a geometric scene representation for depth synthesis from novel viewpoints, including estimation, interpolation, and extrapolation.…”