Dynamic Object Removal and Spatio-Temporal RGB-D Inpainting via Geometry-Aware Adversarial Learning

Abstract

Dynamic objects have a significant impact on the robot's perception of the environment which degrades the performance of essential tasks such as localization and mapping. In this work, we address this problem by synthesizing plausible color, texture and geometry in regions occluded by dynamic objects. We propose the novel geometry-aware DynaFill architecture that follows a coarse-to-fine topology and incorporates our gated recurrent feedback mechanism to adaptively fuse information from previous timesteps. We optimize our architecture using adversarial training to synthesize fine realistic textures which enables it to hallucinate color and depth structure in occluded regions online in a spatially and temporally coherent manner, without relying on future frame information. Casting our inpainting problem as an image-to-image translation task, our model also corrects regions correlated with the presence of dynamic objects in the scene, such as shadows or reflections. We introduce a large-scale hyperrealistic dataset with RGB-D images, semantic segmentation labels, camera poses as well as ground truth RGB-D information of occluded regions. Extensive quantitative and qualitative evaluations show that our approach achieves state-of-the-art performance, even in challenging weather conditions. Furthermore, we present results for retrieval-based visual localization with the synthesized images that demonstrate the utility of our approach.

Unsupervised Domain Adaptation for LiDAR Panoptic Segmentation

“…Autonomous vehicles (AVs) rely on accurate semantic understanding of their surroundings for reliable and safe operation. Scene segmentation is extensively used in various applications such as dynamic object removal [1] and localization [2] as it enables distinguishing points that belong to different objects and classes. It can be classified into three tasks, namely, semantic segmentation which predicts a class label for each point, instance segmentation which assigns a unique ID to points belonging to each object, and panoptic segmentation which combines both semantic and instance segmentation to yield a holistic output containing both stuff and thing classes.…”

DynaSLAM II: Tightly-Coupled Multi-Object Tracking and SLAM

“…The biggest part of the literature tackles this problem by detecting moving regions within the observed scene and rejecting such areas for the SLAM problem [4]- [7]. Some works process the image streams outside of the localization pipeline by translating the images that show dynamic content into realistic images with only static content [8], [9]. On the other hand, a small but growing part of the robotics community has addressed this issue by incorporating the dynamics of moving objects into the problem [10]- [13].…”