RigidFusion: RGB‐D Scene Reconstruction with Rigidly‐moving Objects

Abstract

Figure 1: Given a dynamic scene with rigidly moving objects, RigidFusion performs 4D reconstruction from RGB-D frames (left) and outputs camera motion (shown in green curves), fused object geometries (rendered with light blue and golden yellow), and their respective trajectories (shown in brown/purple curves). Two novel-view reconstructions from two time steps are shown on the middle panel, with frame numbers F i corresponding to different time steps.

Factored Neural Representation for Scene Understanding

“…For both i MAP and N ice SLAM, we employ the open‐source network implementation [ZPL*22] in our training framework instead of their multi‐threads SLAM framework, which contains several optimizations (e.g., view‐purging) for real‐time applications. Note that i MAP, with provided background and object segmentation information, can be seen as an upper bound for performance of a method like RigidFusion [WLNM21].…”

“…Aggregating raw scans while simultaneously estimating and accounting for underlying camera motion is an established way of acquiring large‐scale geometry of rigid scenes (e.g., KinectFusion [IKH*11], VoxelHash [NZIS13]). This paradigm has been extended for dynamic scenes by simultaneously segmenting and tracking multiple (rigid) objects (e.g., CoFusion [RA17], MaskFusion [RBA18], MidFusion [XLT*19], EmFusion [SS19], RigidFusion [WLNM21]) or, decoupling the handling of objects and human motion (e.g., Mixed‐Fusion [ZX17]). These methods explicitly track and represent geometry, without or with textured colors, do not support joint optimization, and need special handling for multiple objects.…”

“…We compare to competing methods and show that ours can produce better object representations and camera/object trajectories. Note that prior methods often focus on only rigid motion and separated optimization [WLNM21, MWM*21], assume access to geometric priors [MWM*21], a single non‐rigid object with local motion [PSH*21,CFF*22], a global representation [LNSW21,TTG*21, XHKK21], foreground‐background separation and novel‐view rendering without geometric reconstruction [GSKH21, YLSL21, WZT*22, SCL*23], or static scenes with an implicit representation [ZPL*22,SLOD21, YPN*22]. We relax many of these restrictions and demonstrate that our factorized representation naturally enables edits involving object‐level manipulations.…”

Mixed Reality Communication for Medical Procedures: Teaching the Placement of a Central Venous Catheter

“…Most recent algorithms utilize machine learning on large data sets [4,10] for high quality reconstruction of RGBD camera [2,3,16,35] footage. Yet, high-quality reconstruction algorithms are too slow [37] for real-time communication. To overcome this issue, real-time 3D surface reconstruction has been proposed [8,15,38].…”