DeepVoxels: Learning Persistent 3D Feature Embeddings

Sitzmann, Vincent; Thies, Justus; Heide, Felix; NieBner, Matthias; Wetzstein, Gordon; Zollhöfer, Michael

doi:10.1109/cvpr.2019.00254

Cited by 504 publications

(395 citation statements)

References 47 publications

Supporting

Mentioning

395

Contrasting

Order By: Relevance

“…Numerous recent works employed DR for learning based 3D vision tasks, such as single view image reconstruction [Pontes et al 2017;Vogels et al 2018;Yan et al 2016;, face reconstruction [Richardson et al 2017], shape completion [Hu et al 2019], and image synthesis [Sitzmann et al 2018]. To describe a few, Pix2Scene [Rajeswar et al 2018] uses a point based DR to learn implicit 3D representations from images.…”

Section: Differentiable Renderingmentioning

confidence: 99%

Differentiable surface splatting for point-based geometry processing

et al. 2019

View full text Add to dashboard Cite

Fig. 1. Using our differentiable point-based renderer, scene content can be optimized to match target rendering. Here, the positions and normals of points are optimized in order to reproduce the reference rendering of the Stanford bunny. It successfully deforms a sphere to a target bunny model, capturing both large scale and fine-scale structures. From left to right are the input points, the results of iteration 18, 57, 198, 300, and the target.We propose Differentiable Surface Splatting (DSS), a high-fidelity differentiable renderer for point clouds. Gradients for point locations and normals are carefully designed to handle discontinuities of the rendering function. Regularization terms are introduced to ensure uniform distribution of the points on the underlying surface. We demonstrate applications of DSS to inverse rendering for geometry synthesis and denoising, where large scale topological changes, as well as small scale detail modifications, are accurately and robustly handled without requiring explicit connectivity, outperforming state-of-the-art techniques. The data and code are at https://github.com/yifita/DSS.

show abstract

Section: Differentiable Renderingmentioning

confidence: 99%

Differentiable surface splatting for point-based geometry processing

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Novel view synthesis is typically solved using image based rendering techniques [Kang et al 2006], with recent approaches allowing for high-quality view synthesis results [Chaurasia et al 2013[Chaurasia et al , 2011Hedman et al 2017;Hedman and Kopf 2018;Penner and Zhang 2017]. With the emergence of deep neural networks, learning-based techniques have become an increasingly popular tool for novel view synthesis [Flynn et al 2016;Ji et al 2017;Kalantari et al 2016;Meshry et al 2019;Mildenhall et al 2019;Sitzmann et al 2019;Thies et al , 2018Zhou et al 2018]. To enable high-quality synthesis results, existing methods typically require multiple input views [Kang et al 2006;Penner and Zhang 2017].…”

Section: Related Work 21 Novel View Synthesismentioning

confidence: 99%

3D Ken Burns effect from a single image

et al. 2019

View full text Add to dashboard Cite

from to Fig. 1. 3D Ken Burns effect from a single image. Given a single input image and optional user annotations in form of two cropping windows, our framework animates the input image while adding parallax to synthesize a 3D Ken Burns effect. Our method works well for a wide variety of content, including portrait (top) and landscape (bottom) photos. Please refer to our supplementary video demo for more examples. Please note that this figure, as well as many other figures in this paper, contain video clips. Should these videos not already be playing then please consider viewing this paper using Adobe Reader.The Ken Burns effect allows animating still images with a virtual camera scan and zoom. Adding parallax, which results in the 3D Ken Burns effect, enables significantly more compelling results. Creating such effects manually is timeconsuming and demands sophisticated editing skills. Existing automatic methods, however, require multiple input images from varying viewpoints. In this paper, we introduce a framework that synthesizes the 3D Ken Burns effect from a single image, supporting both a fully automatic mode and an interactive mode with the user controlling the camera. Our framework first leverages a depth prediction pipeline, which estimates scene depth that is suitable for view synthesis tasks. To address the limitations of existing depth estimation methods such as geometric distortions, semantic distortions, and inaccurate depth boundaries, we develop a semantic-aware neural network for depth prediction, couple its estimate with a segmentation-based depth adjustment process, and employ a refinement neural network that facilitates accurate depth predictions at object boundaries. According to this depth estimate, our framework then maps the input image to a point cloud and synthesizes the resulting video frames by rendering the point cloud from the corresponding camera positions. To address disocclusions while maintaining geometrically and temporally coherent synthesis results, we utilize contextaware color-and depth-inpainting to fill in the missing information in the extreme views of the camera path, thus extending the scene geometry of the point cloud. Experiments with a wide variety of image content show that our method enables realistic synthesis results. Our study demonstrates that our system allows users to achieve better results while requiring little effort compared to existing solutions for the 3D Ken Burns effect creation.

show abstract

“…DeepVoxels [Sitzmann et al 2018] automatically learns a 3D feature representation for novel view synthesis but is limited to static objects and scenes, and the employed architecture does not lend itself to real-time inference. Martin-Brualla et al [2018] use neural networks to fill holes and generally improve the quality of a textured geometric representation.…”

Section: Neural Renderingmentioning

confidence: 99%

Neural volumes

et al. 2019

View full text Add to dashboard Cite

Fig. 1. Renderings of objects captured and modeled by our system. The input to our method consists of synchronized and calibrated multi-view video. We build a dynamic, volumetric representation of the scene by training an encoder-decoder network end-to-end using a differentiable ray marching algorithm.Modeling and rendering of dynamic scenes is challenging, as natural scenes often contain complex phenomena such as thin structures, evolving topology, translucency, scattering, occlusion, and biological motion. Mesh-based reconstruction and tracking often fail in these cases, and other approaches (e.g., light field video) typically rely on constrained viewing conditions, which limit interactivity. We circumvent these difficulties by presenting a learningbased approach to representing dynamic objects inspired by the integral projection model used in tomographic imaging. The approach is supervised directly from 2D images in a multi-view capture setting and does not require explicit reconstruction or tracking of the object. Our method has two primary components: an encoder-decoder network that transforms input images into a 3D volume representation, and a differentiable ray-marching operation that enables end-to-end training. By virtue of its 3D representation, our construction extrapolates better to novel viewpoints compared to screen-space rendering techniques. The encoder-decoder architecture learns a latent representation of a dynamic scene that enables us to produce novel content sequences not seen during training. To overcome memory limitations of voxel-based representations, we learn a dynamic irregular grid structure implemented with a warp field during ray-marching. This structure greatly improves the apparent resolution and reduces grid-like artifacts and jagged motion. Finally, we demonstrate how to incorporate surface-based representations into our volumetric-learning framework for applications

show abstract

DeepVoxels: Learning Persistent 3D Feature Embeddings

Cited by 504 publications

References 47 publications

Differentiable surface splatting for point-based geometry processing

Differentiable surface splatting for point-based geometry processing

3D Ken Burns effect from a single image

Neural volumes

Contact Info

Product

Resources

About