Adop

Abstract

In this paper we present ADOP, a novel point-based, differentiable neural rendering pipeline. Like other neural renderers, our system takes as input calibrated camera images and a proxy geometry of the scene, in our case a point cloud. To generate a novel view, the point cloud is rasterized with learned feature vectors as colors and a deep neural network fills the remaining holes and shades each output pixel. The rasterizer renders points as one-pixel splats, which makes it very fast and allows us to compute gradients with respect to all relevant input parameters efficiently. Furthermore, our pipeline contains a fully differentiable physically-based photometric camera model, including exposure, white balance, and a camera response function. Following the idea of inverse rendering, we use our renderer to refine its input in order to reduce inconsistencies and optimize the quality of its output. In particular, we can optimize structural parameters like the camera pose, lens distortions, point positions and features, and a neural environment map, but also photometric parameters like camera response function, vignetting, and per-image exposure and white balance. Because our pipeline includes photometric parameters, e.g. exposure and camera response function, our system can smoothly handle input images with varying exposure and white balance, and generates high-dynamic range output. We show that due to the improved input, we can achieve high render quality, also for difficult input, e.g. with imperfect camera calibrations, inaccurate proxy geometry, or varying exposure. As a result, a simpler and thus faster deep neural network is sufficient for reconstruction. In combination with the fast point rasterization, ADOP achieves real-time rendering rates even for models with well over 100M points. https://github.com/darglein/ADOP

Advances in Neural Rendering

“…In Neural Point‐based Graphics [ASK*20a] and SynSin [WGSJ20], learnable features are attached to the points that can store rich information about the appearance and shape of the actual surface. In ADOP [RFS21a] these learnable features are interpreted by an MLP which can account for view‐dependent effects. Note that instead of storing explicitly features for specific points, one can also use an MLP to predict the features for the discrete positions.…”

Cross-task feature alignment for seeing pedestrians in the dark

“…Thies et al [36] incorporate neural textures into traditional mesh rasterization pipeline and use a CNN-based neural renderer to enable high quality NVS. Another line of approaches [2,18,27,31,53] follow a similar pipeline but use point as the surface primitive and directly featurize the surface in 3D. Recently, 3D Gaussian Splatting [16] demonstrates great success in terms of rendering quality and efficiency with a highly flexible point-based representation.…”

“…Secondly, there lacks a suitable representation simultaneously possessing global robustness and local expressivity. Formulations that favor the former typically rely on an imperfect surface proxy reconstructed from multiview stereo (MVS) [10,15,32,46], and then featurize the proxy on either the parametrized 2D UV map [23,36,39] or the 3D surface [2,16,18,27,31,45,53]. While being global-structure-aware, such representations typically struggle to represent the local intricate details.…”

“…1 (a), UV-based featurizations [23,36,39] suffer from the inherent distortion of surface parametrization [8,14,28], which severely degrades the rendering quality. Meanwhile, representations based on more global 3D discretization, such as point cloud [2,18,27,31,53], mesh [45], or Gaussians [16], all suffer from limited featuremetric resolutions, i.e., the resolutions of the local spatial features, which depend on the respective discretization resolution. As shown in Fig.…”

Frequency-Modulated Point Cloud Rendering with Easy Editing

“…In contrast to implicit rendering, point cloud rendering [1,6,13,18,33,36,59] is a promising editable rendering model. On the one hand, explicit 3D representations are better for interactive editing.…”