Towards Complete Scene Reconstruction from Single-View Depth and Human Motion

Abstract

Complete scene reconstruction from single view RGBD is a challenging task, requiring estimation of scene regions occluded from the captured depth surface. We propose that scene-centric analysis of human motion within an indoor scene can reveal fully occluded objects and provide functional cues to enhance scene understanding tasks. Captured skeletal joint positions of humans, utilised as naturally exploring active sensors, are projected into a human-scene motion representation. Inherent body occupancy is leveraged to carve a volumetric scene occupancy map initialised from captured depth, revealing a more complete voxel representation of the scene. To obtain a structured box model representation of the scene, we introduce unique terms to an object detection optimisation that overcome depth occlusions whilst deriving from the same depth data. The method is evaluated on challenging indoor scenes with multiple occluding objects such as tables and chairs. Evaluation shows that human-centric scene analysis can be applied to effectively enhance state-of-the-art scene understanding approaches, resulting in a more complete representation than single view depth alone.

Human-Centric Scene Understanding from Single View 360 Video

“…The results show that introducing the scene affordance map to the reconstruction process improves both the precision and recall of occupied voxels. Note that [8] Table 3. Affordance segmentation results on synthetic and real scenes.…”

“…Real-person analysis has been applied outside, in applications such as surveillance monitoring: Rother et al [23] demonstrate deriving 3D information from tracked people to determine scene geometry whilst Ballen et al [3] learn navigation paths to predict future activities and semantically segment scenes. Indoors, Fowler et al [8] expand Fouhey et al's [7] concept of providing unique cues from activity analysis to improve standard 3D reconstruction approaches, whilst Delaitre et al [5] study semantic object recognition from long-term observation of human-object interactions.…”

“…The reconstruction technique applies scene structure constraints on the predicted affordance segmentation, alongside the assumption discussed in [8] that space occupied by a human body must be free space. The reconstruction is produced solely by applying these constraints to M act and M aff to estimate a 3D voxel grid, with no current consideration of visual scene information.…”

“…The proposed reconstruction approach is evaluated on the synthetic and real datasets to validate the approach against the state of the art [8], assessing how the introduction of affordance prediction improves reconstruction over applying only human motion. Four variants of the proposed method are evaluated against [8] in an ablation study: our technique applying only the ground truth affordance map (Ours-GT-Aff ) and applying it with the activity map (Ours-GT-Aff+Act), and our technique applying only the predicted affordance map (Ours-Pred-Aff ) and applying it with the activity map (Ours-Pred-Aff+Act).…”

“…Voxel-level precision, recall and intersection over union (IoU) [27] are applied as the evaluation metric; all reconstructed voxels are treated as one class as [8] has no semantic segmentation component. Only voxels within the ground truth 3D reconstruction are considered in the evaluation.…”