Neural3D: Light-weight Neural Portrait Scanning via Context-aware Correspondence Learning

Suo, Xin; Wu, Minye; Zhang, Yanshun; Zhang, Yingliang; Xu, Lan; Hu, Qiang; Yu, Jingyi

doi:10.1145/3394171.3413734

Cited by 6 publications

(3 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the area of photo- realistic novel view synthesis and 3D scene reconstruction, neural rendering shows great power and huge potential. Various data representations are adopted to obtain better performance and characteristics, such as pointclouds [1,48,54], voxels [28,28], texture meshes [26,51] or implicit functions [31,34] and hybrid neural blending [46,47]. NHR [54] embeds spatial features into sparse dynamic point-clouds, Neural Volumes [28] transforms input images into a 3D volume representation by a VAE network.…”

Section: Related Workmentioning

confidence: 99%

NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions

Jiang¹,

Jiang²,

Sun³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

4D modeling of human-object interactions is critical for numerous applications. However, efficient volumetric capture and rendering of complex interaction scenarios, especially from sparse inputs, remain challenging. In this paper, we propose NeuralFusion, a neural approach for volumetric human-object capture and rendering using sparse consumer RGBD sensors. It marries traditional non-rigid fusion with recent neural implicit modeling and blending advances, where the captured humans and objects are layerwise disentangled. For geometry modeling, we propose a neural implicit inference scheme with non-rigid key-volume fusion, as well as a template-aid robust object tracking pipeline. Our scheme enables detailed and complete geometry generation under complex interactions and occlusions. Moreover, we introduce a layer-wise human-object texture rendering scheme, which combines volumetric and image-based rendering in both spatial and temporal domains to obtain photo-realistic results. Extensive experiments demonstrate the effectiveness and efficiency of our approach in synthesizing photo-realistic free-view results under complex human-object interactions.

show abstract

Section: Related Workmentioning

confidence: 99%

NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions

Jiang¹,

Jiang²,

Sun³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the photo-realistic novel view synthesis and 3D scene modeling domain, differentiable neural rendering based on various data proxies achieves impressive results and becomes more and more popular. Various data representations are adopted to obtain better performance and characteristics, such as point-clouds [Aliev et al 2020;Suo et al 2020;], voxels [Lombardi et al 2019], texture meshes Shysheya et al 2019;Thies et al 2019] or implicit functions [Kellnhofer et al 2021;Mildenhall et al 2020;Park et al 2019] and hybrid neural blending [Jiang et al 2022a;Sun et al 2021;Suo et al 2021]. More recently, [Li et al 2020;Park et al 2020;Pumarola et al 2021;] extend neural radiance field [Mildenhall et al 2020] into the dynamic setting.…”

Section: Related Workmentioning

confidence: 99%

Human Performance Modeling and Rendering via Neural Animated Mesh

Zhao¹,

Jiang²,

Yao³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

We have recently seen tremendous progress in the neural advances for photo-real human modeling and rendering. However, it's still challenging to integrate them into an existing mesh-based pipeline for downstream applications. In this paper, we present a comprehensive neural approach for high-quality reconstruction, compression, and rendering of human performances from dense multi-view videos. Our core intuition is to bridge the traditional animated mesh workflow with a new class of highly efficient neural techniques. We first introduce a neural surface reconstructor for highquality surface generation in minutes. It marries the implicit volumetric for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

show abstract

“…The recent progress of differentiable neural rendering brings huge potential for 3D scene modeling and photorealistic novel view synthesis. Researchers explore various data representations to pursue better performance and characteristics, such as point-clouds [2,58,64], voxels [31], texture meshes [27,60] or implicit functions [7,33,34,36,43,63]. However, these methods…”

Section: Related Workmentioning

confidence: 99%

Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions

Sun

Chen

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

human-object rendering scheme, which combines direction-aware neural blending weight learning and spatial-temporal texture completion to provide high-resolution and photo-realistic texture results in the occluded scenarios. Extensive experiments demonstrate the effectiveness of our approach to achieve high-quality geometry and texture reconstruction in free viewpoints for challenging human-object interactions. CCS CONCEPTS• Computing methodologies → Image-based rendering.

show abstract

Neural3D: Light-weight Neural Portrait Scanning via Context-aware Correspondence Learning

Abstract: Figure 1: Illustration of our Neural3D system, which achieves convenient and realistic neural reconstruction and freeviewpoint rendering of human portraits from only a single portable RGB camera.

Cited by 6 publications

References 47 publications

NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions

NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions

Human Performance Modeling and Rendering via Neural Animated Mesh

Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions

Contact Info

Product

Resources

About