ADOP: Approximate Differentiable One-Pixel Point Rendering

Rückert, Darius; Franke, Linus; Stamminger, Marc

doi:10.48550/arxiv.2110.06635

Cited by 20 publications

(31 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent advances in image-based rendering include learning-based disparity estimation [9,16], blending [14,39], and image synthesis via 2D CNNs [39,40]. Other commonly used scene representations include voxels [25,47], 3D point clouds [1,18,41] or cameracentric layered 3D representations, such as multi-plane images [3,29,53,57,64] or layered depth images [13,19,43,54].…”

Section: Related Workmentioning

confidence: 99%

Learning Neural Light Fields with Ray-Space Embedding Networks

Attal¹,

Huang²,

Zollhoefer³

et al. 2021

Preprint

View full text Add to dashboard Cite

Neural radiance fields (NeRFs) produce state-of-the-art view synthesis results. However, they are slow to render, requiring hundreds of network evaluations per pixel to approximate a volume rendering integral. Baking NeRFs into explicit data structures enables efficient rendering, but results in a large increase in memory footprint and, in many cases, a quality reduction. In this paper, we propose a novel neural light field representation that, in contrast, is compact and directly predicts integrated radiance along rays. Our method supports rendering with a single network evaluation per pixel for small baseline light field datasets and can also be applied to larger baselines with only a few evaluations per pixel. At the core of our approach is a ray-space embedding network that maps the 4D ray-space manifold into an intermediate, interpolable latent space. Our method achieves state-ofthe-art quality on dense forward-facing datasets such as the Stanford Light Field dataset. In addition, for forwardfacing scenes with sparser inputs we achieve results that are competitive with NeRF-based approaches in terms of quality while providing a better speed/quality/memory trade-off with far fewer network evaluations. * This work was done while Benjamin was an intern at Meta.

show abstract

Section: Related Workmentioning

confidence: 99%

Learning Neural Light Fields with Ray-Space Embedding Networks

Attal¹,

Huang²,

Zollhoefer³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Multi-plane images [2,5,35,57,77] are good at modeling continuous shapes similar to volumetric representations, but are limited to a constrained set of viewing angles. Point cloud representations [1,20,34,50,66] can model various geometries with high fidelity. When used for appearance modeling, however, point-based representations might suffer from their innate sparseness which might result in holes.…”

Section: Related Workmentioning

confidence: 99%

“…They have achieved impressive results in 3D scene acquisition and rendering from multi-view photometric information. Compared to other geometric representations like multi-plane images [2,5,35,57,77] or point-based representations [1,20,34,50,66], volumetric representations support a larger range of camera motion for view extrapolation and do not suffer from holes when rendering dynamic geometry like point-based representations. Furthermore, they can be learned from multi-view RGB data using differentiable volumetric ray marching, without additional MVS methods.…”

Section: Introductionmentioning

confidence: 99%

HVH: Learning a Hybrid Neural Volumetric Representation for Dynamic Hair Performance Capture

Wang¹,

Nam²,

Stuyck³

et al. 2021

Preprint

View full text Add to dashboard Cite

Capturing and rendering life-like hair is particularly challenging due to its fine geometric structure, the complex physical interaction and its non-trivial visual appearance. Yet, hair is a critical component for believable avatars. In this paper, we address the aforementioned problems: 1) we use a novel, volumetric hair representation that is composed of thousands of primitives. Each primitive can be rendered efficiently, yet realistically, by building on the latest advances in neural rendering. 2) To have a reliable control signal, we present a novel way of tracking hair on the strand level. To keep the computational effort manageable, we use guide hairs and classic techniques to expand those into a dense hood of hair. 3) To better enforce temporal consistency and generalization ability of our model, we further optimize the 3D scene flow of our representation with multiview optical flow, using volumetric raymarching. Our method can not only create realistic renders of recorded multi-view sequences, but also create renderings for new hair configurations by providing new control signals. We compare our method with existing work on viewpoint synthesis and drivable animation and achieve stateof-the-art results. Please check out our project website at https://ziyanw1.github.io/hvh/.

show abstract

“…Representing shape and appearance of scenes using neural networks has recently gained immense popularity. Explicit representation-based methods use differentiable rendering to learn 3D representation such as point clouds [1,41,58], meshes [51] or voxels [29,44] for the scene. Implicit representation-based methods represent scenes using continuous coordinatebased functions such as signed distance fields [2,8,17,24,60] or occupancy fields [30,35].…”

Section: Related Workmentioning

confidence: 99%

Light Field Neural Rendering

Suhail¹,

Esteves²,

Sigal³

et al. 2021

Preprint

View full text Add to dashboard Cite

Classical light field rendering for novel view synthesis can accurately reproduce view-dependent effects such as reflection, refraction, and translucency, but requires a dense view sampling of the scene. Methods based on geometric reconstruction need only sparse views, but cannot accurately model non-Lambertian effects. We introduce a model that combines the strengths and mitigates the limitations of these two directions. By operating on a four-dimensional representation of the light field, our model learns to represent view-dependent effects accurately. By enforcing geometric constraints during training and inference, the scene geometry is implicitly learned from a sparse set of views. Concretely, we introduce a two-stage transformer-based model that first aggregates features along epipolar lines, then aggregates features along reference views to produce the color of a target ray. Our model outperforms the state-ofthe-art on multiple forward-facing and 360 • datasets, with larger margins on scenes with severe view-dependent variations. Code and results can be found at light-field-neuralrendering.github.io.* Work done while interning at Google. Ground TruthOurs (35.3 dB) NeX (30.4 dB) NeRF (29.6 dB)

show abstract

ADOP: Approximate Differentiable One-Pixel Point Rendering

Cited by 20 publications

References 0 publications

Learning Neural Light Fields with Ray-Space Embedding Networks

Learning Neural Light Fields with Ray-Space Embedding Networks

HVH: Learning a Hybrid Neural Volumetric Representation for Dynamic Hair Performance Capture

Light Field Neural Rendering

Contact Info

Product

Resources

About