Real-Time High-Resolution Background Matting

Lin, Shanchuan; Ryabtsev, Andrey; Sengupta, Shamik; Curless, Brian; Seitz, Steve; Kemelmacher-Shlizerman, Ira

doi:10.1109/cvpr46437.2021.00865

Cited by 186 publications

(118 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The captured RGB resolution is 2560 × 1440 and depth image resolution 1024 × 1024. During testing, we use background-matting-v2 [33] to obtain the mask M of body portraits and then use RetinaFace [9] to detect the front face for the front-view. It takes about 3.57s for our model (32-bit floating-point precision) to predict the occupancy field of resolution 256 with the given inputs.…”

Section: Methodsmentioning

confidence: 99%

“…3. For clarity, we denote RGB body portrait as I, the corresponding binary body mask as M obtained by the background-matting method [33], D raw as the normalized result of the captured depth map, and the refined depth map as D rf , where D rf = DRM(D raw , I). The architecture with two different types of inputs, i.e., I and D rf , is inspired by the hypercolumn structure in [20] and DDR-Net [60].…”

Section: Depth Refinementmentioning

confidence: 99%

See 1 more Smart Citation

Total Scale: Face-to-Body Detail Reconstruction from Sparse RGBD Sensors

Zheng¹,

Xu²,

Duan³

et al. 2021

Preprint

View full text Add to dashboard Cite

While the 3D human reconstruction methods using Pixelaligned implicit function (PIFu) develop fast, we observe that the quality of reconstructed details is still not satisfactory. Flat facial surfaces frequently occur in the PIFubased reconstruction results. To this end, we propose a twoscale PIFu representation to enhance the quality of the reconstructed facial details. Specifically, we utilize two MLPs to separately represent the PIFus for the face and human body. An MLP dedicated to the reconstruction of 3D faces can increase the network capacity and reduce the difficulty of the reconstruction of facial details as in the previous onescale PIFu representation. To remedy the topology error, we leverage 3 RGBD sensors to capture multiview RGBD data as the input to the network, a sparse, lightweight capture setting. Since the depth noise severely influences the reconstruction results, we design a depth refinement module to reduce the noise of the raw depths under the guidance of the input RGB images. We also propose an adaptive fusion scheme to fuse the predicted occupancy field of the body and face to eliminate the discontinuity artifact at their boundaries. Experiments demonstrate the effectiveness of our approach in reconstructing vivid facial details and deforming body shapes, and verify its superiority over state-of-the-art methods.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Depth Refinementmentioning

confidence: 99%

Total Scale: Face-to-Body Detail Reconstruction from Sparse RGBD Sensors

Zheng¹,

Xu²,

Duan³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Due to the nature of relying on low-level color cues, their assumptions are easily violated in complex images. To overcome this dilemma, deep matting methods [1,4,11,14,18,19,22,28,29,34,36] appeared with the development of deep learning.…”

Section: Related Workmentioning

confidence: 99%

Boosting Robustness of Image Matting with Context Assembling and Strong Data Augmentation

Dai¹,

Price²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

SIM GCA IndexNet FBA CA+DA Ours Figure 1. Matting results on real-world images. From the second column to right are results of IndexNet [22], GCA [18], SIM [28], FBA [11], CA+data augmentation (DA) [14] and our method, respectively. Note that, all the methods are trained with the DIM [34]dataset (except SIM is trained with the SIMD [28] dataset). They are comparable on benchmark images, while presents varying results on real-world images. Our method shows better generalization ability.

show abstract

“…The input videos contain background, which we do not want to reconstruct. We obtain foreground segmentations for all input images via image matting [26] together with a hard brightness threshold. During training, we use a background loss L background to discourage geom-etry along rays of background pixels.…”

Section: D Reconstructionmentioning

confidence: 99%

Virtual Elastic Objects

Chen¹,

Tretschk²,

Stuyck³

et al. 2022

Preprint

View full text Add to dashboard Cite

Figure 1. Method overview. We use a multi-view capture system to record objects deforming under the influence of external forces. Our method reconstructs a meshless geometry and deformation field from these sequences. Using a differentiable simulator, we optimize the material parameters of the objects to match the observations. These parameters allow us to find novel, plausible object configurations in response to new forces fields or collision constraints due to user interactions Finally, we re-render the deformed state.

show abstract

Real-Time High-Resolution Background Matting

Cited by 186 publications

References 27 publications

Total Scale: Face-to-Body Detail Reconstruction from Sparse RGBD Sensors

Total Scale: Face-to-Body Detail Reconstruction from Sparse RGBD Sensors

Boosting Robustness of Image Matting with Context Assembling and Strong Data Augmentation

Virtual Elastic Objects

Contact Info

Product

Resources

About