Multi-View Neural Human Rendering

Wu, Minye; Wang, Yuehao; Hu, Qiang; Yu, Jingyi

doi:10.1109/cvpr42600.2020.00175

Cited by 99 publications

(84 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The free-view results, however, are vulnerable to occlusions and suffer from limited view interpolation along with the dense captured views, leading to uncanny texture details due to view blending. The recent neural rendering techniques bring huge potential for compelling photo-realistic free-viewpoint video generation via neural view blending [Meshry et al 2019a;Thies et al 2018] or neural scene modeling [Bemana et al 2020;Mildenhall et al 2020a;Wu et al 2020b]. Open4D [Bansal et al 2020] generates a free-viewpoint video enabling occlusion removal and time-freezing effects using around 15 mobile cameras, which is similar to our work.…”

Section: Introductionsupporting

confidence: 74%

“…However, this method suffers from limited free-viewing range and fragile dynamic motion modeling. For reconstructing neural scenes, various data representations have been explored, such as point-clouds [Aliev et al 2020;Wu et al 2020b], voxels [Lombardi et al 2019;Sitzmann et al 2019a] or implicit representations [Mildenhall et al 2020a;Sitzmann et al 2019b;Suo et al 2021]. Researchers [Jin et al 2018;Kwon et al 2020] also utilize the underlying latent geometry for novel view synthesis of human performance in the encoder-decoder manner, which still suffers from limited representation ability of a single latent code for complex human inferior texture output.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Editable free-viewpoint video using a layered neural representation

Zhang

Liu

et al. 2021

ACM Trans. Graph.

Self Cite

View full text Add to dashboard Cite

Generating free-viewpoint videos is critical for immersive VR/AR experience, but recent neural advances still lack the editing ability to manipulate the visual perception for large dynamic scenes. To fill this gap, in this paper, we propose the first approach for editable free-viewpoint video generation for large-scale view-dependent dynamic scenes using only 16 cameras. The core of our approach is a new layered neural representation, where each dynamic entity, including the environment itself, is formulated into a spatiotemporal coherent neural layered radiance representation called ST-NeRF. Such a layered representation supports manipulations of the dynamic scene while still supporting a wide free viewing experience. In our ST-NeRF, we represent the dynamic entity/layer as a continuous function, which achieves the disentanglement of location, deformation as well as the appearance of the dynamic entity in a continuous and self-supervised manner. We propose a scene parsing 4D label map tracking to disentangle the spatial information explicitly and a continuous deform module to disentangle the temporal motion implicitly. An object-aware volume rendering scheme is further

show abstract

Section: Introductionsupporting

confidence: 74%

Section: Related Workmentioning

confidence: 99%

Editable free-viewpoint video using a layered neural representation

Zhang

Liu

et al. 2021

ACM Trans. Graph.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The results in Table 1 verify our advanced generalization capacity on the unseen scenarios. We also achieve competitive fitting performance on the training frames, even comparable to the per-scene optimization methods [28,37,39].…”

Section: Synthesis Performance Analysismentioning

confidence: 93%

“…Recently, researchers adopt neural networks to represent the shape and appearance of scenes. These representations, such as voxels [19,21,26,33], point clouds [1,39], textured meshes [14,16,18,40] and multi-plane images [7,44] are learned from 2D images via differentiable renderers. Though with impressive results, they are hard to scale to higher resolution due to innate cubic memory complexity.…”

Section: Related Workmentioning

confidence: 99%

“…We train our model on 7 random human bodies from the THUman dataset (THUman-7) and all 202 human bodies (THUman-all) separately, and test the synthesis performance on the test frames of ZJU-3. From [19] ZJU-3 ZJU-3 20.84 0.827 NT [37] ZJU-3 ZJU-3 21.92 0.873 NHR [39] ZJU-3 ZJU-3 22.03 0.875 NB [28] ZJU-3 ZJU-3 22.88 0.880 PVA [30] ZJU-7 ZJU-3 23.15 0.866 Pixel-NeRF [41] ZJU-7 ZJU-3 23.17 0.869 NHP [12] ZJU-7 ZJU-3 24.75 0.906 GP-NeRF (Ours) ZJU-7 ZJU-3 25.96 0.921 Generalization performance across datasets NHP [12] AIST ZJU-3 17.05 0.771 GP-NeRF (Ours) THUman-7 ZJU-3 24.74 0.907 GP-NeRF (Ours) THUman-all ZJU-3 25.60 0.917 Table 1. Synthesis performance comparison.…”

Section: Synthesis Performance Analysismentioning

confidence: 99%

See 1 more Smart Citation

Geometry-Guided Progressive NeRF for Generalizable and Efficient Neural Human Rendering

Chen¹,

Zhang²,

Xu³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this work we develop a generalizable and efficient Neural Radiance Field (NeRF) pipeline for high-fidelity free-viewpoint human body synthesis under settings with sparse camera views. Though existing NeRF-based methods can synthesize rather realistic details for human body, they tend to produce poor results when the input has self-occlusion, especially for unseen humans under sparse views. Moreover, these methods often require a large number of sampling points for rendering, which leads to low efficiency and limits their real-world applicability. To address these challenges, we propose a Geometry-guided Progressive NeRF (GP-NeRF). In particular, to better tackle self-occlusion, we devise a geometry-guided multi-view feature integration approach that utilizes the estimated geometry prior to integrate the incomplete information from input views and construct a complete geometry volume for the target human body. Meanwhile, for achieving higher rendering efficiency, we introduce a geometry-guided progressive rendering pipeline, which leverages the geometric feature volume and the predicted density values to progressively reduce the number of sampling points and speed up the rendering process. Experiments on the ZJU-MoCap and THUman datasets show that our method outperforms the state-of-the-arts significantly across multiple generalization settings, while the time cost is reduced via > 70% via applying our efficient progressive rendering pipeline.(a) Geometry-guided image feature integration: ours (left) vs. previous (right). V for view and T for time.

show abstract