SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video

Abstract

Figure 1. We propose MultiPly, a novel framework to reconstruct multiple people in 3D from in-the-wild monocular videos. Our method can recover the complete 3D human with high-fidelity shape and appearance, even in scenarios involving occlusions and close interactions.

Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video

“…To break through the limited representation ability of explicit mesh based digital human representation, many works adopt the implicit representation to improve the model capacity and visual quality [Gafni et al 2021;Jiang et al 2022;Yenamandra et al 2021;Zhuang et al 2021]. i3DMM [Yenamandra et al 2021] is the first neural implicit function based 3D morphable model of full heads.…”

SelfNeRF: Fast Training NeRF for Human from Monocular Self-rotating Video

“…ICON [37] infers a 3D clothed human meshes from a color image by utilizing a body-guided normal prediction model and a local-featurebased implicit 3D representation conditioned on SMPL(-X). SelfRecon [11] represents the human body as a template mesh and SDF in canonical space and utilizes a deformation field consisting of rigid forward LBS deformation and small non-rigid deformation to generate correspondences. Given monocular self-rotation RGB inputs, these methods are ca-pable of generating meshes of clothed humans.…”

“…Although INGP [22] is capable of converging in a short time and rendering high-fidelity novel view images, it is only suitable for static scenes and needs dense multi-view inputs. To aggregate the corresponding point information of different frames, we introduce a surface-relative representation relative to the surface point cloud of the human body (in practice, SMPL [19] or the mesh obtained by Sel-fRecon [11]). We also adopt a multi-resolution hash encoding [22] to speed up training.…”

“…To evaluate our method's reconstruction ability from single-view input, we capture some custom data, which includes monocular videos of human performers and the corresponding high-fidelity template meshes reconstructed by SelfRecon [11]. Each video contains the whole body information of the performer from a single view.…”

“…We achieve these targets via a novel surface-relative hash encoding by extending multi-resolution hash encoding [22] to dynamic objects while aggregating information across frames. Specifically, given the monocular selfrotation video of a human performer, we recover the surface shape for each frame with existing reconstruction methods like VideoAvatar [1] and SelfRecon [11]. We then calculate the K-nearest neighbor points and signed distance on the current frame's point cloud for each query point and take the corresponding point of the k-nearest neighbor points on the canonical space and signed distance as relative representation.…”