Neural Radiance Fields From Sparse RGB-D Images for High-Quality View Synthesis

Yuan, Yujie; Lai, Yu‐Kun; Huang, Yihua; Kobbelt, Leif; Gao, Lin

doi:10.1109/tpami.2022.3232502

Cited by 9 publications

(3 citation statements)

References 84 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…SlimmeRF (Yuan & Zhao, 2023) enhances the TensoRF VM framework by introducing an adaptive rank mechanism, dynamically adjusting the model's learning capacity. The model starts with a low-rank representation and incrementally increases the rank based on learning progress, capturing essential features early and building complexity as needed.…”

Section: J Exploring Rank Incrementation With Slimmerfmentioning

confidence: 99%

Text-Driven Stylization of Video Objects

Loeschcke¹,

Belongie²,

Benaim³

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The ability to learn compact, high-quality, and easy-to-optimize representations for visual data is paramount to many applications such as novel view synthesis and 3D reconstruction. Recent work has shown substantial success in using tensor networks to design such compact and highquality representations. However, the ability to optimize tensor-based representations, and in particular, the highly compact tensor train representation, is still lacking. This has prevented practitioners from deploying the full potential of tensor networks for visual data. To this end, we propose 'Prolongation Upsampling Tensor Train (PuTT)', a novel method for learning tensor train representations in a coarse-to-fine manner. Our method involves the prolonging or 'upsampling' of a learned tensor train representation, creating a sequence of 'coarse-to-fine' tensor trains that are incrementally refined. We evaluate our representation along three axes: (1). compression, (2). denoising capability, and (3). image completion capability. To assess these axes, we consider the tasks of image fitting, 3D fitting, and novel view synthesis, where our method shows an improved performance compared to state-of-the-art tensor-based methods.

show abstract

Section: J Exploring Rank Incrementation With Slimmerfmentioning

confidence: 99%

Text-Driven Stylization of Video Objects

Loeschcke¹,

Belongie²,

Benaim³

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…N OVEL view synthesis has been extensively studied in computer vision and computer graphics. In particular, the recently proposed neural radiance field (NeRF) [1] has inspired a large number of follow-up works aiming to achieve better visual effects [2], faster rendering speed [3], [4], generalization to different scenes [5], relighting [6], [7], applying to dynamic scenes [8], and reducing the number of inputs [9], [10]. However, as an implicit modeling method, the neural radiance field is difficult for users to interactively edit or modify the scene objects, which is relatively easy with explicit representations.…”

Section: Introductionmentioning

confidence: 99%

Interactive NeRF Geometry Editing With Shape Priors

Yuan,

Sun,

Lai

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

View full text Add to dashboard Cite

Neural Radiance Fields (NeRFs) have shown great potential for tasks like novel view synthesis of static 3D scenes. Since NeRFs are trained on a large number of input images, it is not trivial to change their content afterwards. Previous methods to modify NeRFs provide some control but they do not support direct shape deformation which is common for geometry representations like triangle meshes. In this paper, we present a NeRF geometry editing method that first extracts a triangle mesh representation of the geometry inside a NeRF. This mesh can be modified by any 3D modeling tool (we use ARAP mesh deformation). The mesh deformation is then extended into a volume deformation around the shape which establishes a mapping between ray queries to the deformed NeRF and the corresponding queries to the original NeRF. The basic shape editing mechanism is extended towards more powerful and more meaningful editing handles by generating box abstractions of the NeRF shapes which provide an intuitive interface to the user. By additionally assigning semantic labels, we can even identify and combine parts from different objects. We demonstrate the performance and quality of our method in a number of experiments on synthetic data as well as real captured scenes.

show abstract

“…Despite the efficiency of these methods, higher rendering quality cannot be achieved with sparse inputs. Yuan et al [10] enhanced the quality of novel views by reconstructing the scene using depth information and pre-training a model with renderings of the scene. Such a method allows for better novel views under sparse input conditions.…”

Section: Introductionmentioning

confidence: 99%

Prior-Driven NeRF: Prior Guided Rendering

et al. 2023

View full text Add to dashboard Cite

Neural radiation field (NeRF)-based novel view synthesis methods are gaining popularity. NeRF can generate more detailed and realistic images than traditional methods. Conventional NeRF reconstruction of a room scene requires at least several hundred images as input data and generates several spatial sampling points, placing a tremendous burden on the training and prediction process with respect to memory and computational time. To address these problems, we propose a prior-driven NeRF model that only accepts sparse views as input data and reduces a significant number of non-functional sampling points to improve training and prediction efficiency and achieve fast high-quality rendering. First, this study uses depth priors to guide sampling, and only a few sampling points near the controllable range of the depth prior are used as input data, which reduces the memory occupation and improves the efficiency of training and prediction. Second, this study encodes depth priors as distance weights into the model and guides the model to quickly fit the object surface. Finally, a novel approach combining the traditional mesh rendering method (TMRM) and the NeRF volume rendering method was used to further improve the rendering efficiency. Experimental results demonstrated that our method had significant advantages in the case of sparse input views (11 per room) and few sampling points (8 points per ray).

show abstract

Neural Radiance Fields From Sparse RGB-D Images for High-Quality View Synthesis

Cited by 9 publications

References 84 publications

Text-Driven Stylization of Video Objects

Text-Driven Stylization of Video Objects

Interactive NeRF Geometry Editing With Shape Priors

Prior-Driven NeRF: Prior Guided Rendering

Contact Info

Product

Resources

About