Empty Cities: A Dynamic-Object-Invariant Space for Visual SLAM

Abstract

In this paper we present a data-driven approach to obtain the static image of a scene, eliminating dynamic objects that might have been present at the time of traversing the scene with a camera. The general objective is to improve vision-based localization and mapping tasks in dynamic environments, where the presence (or absence) of different dynamic objects in different moments makes these tasks less robust. We introduce an endto-end deep learning framework to turn images of an urban environment that include dynamic content, such as vehicles or pedestrians, into realistic static frames suitable for localization and mapping. This objective faces two main challenges: detecting the dynamic objects, and inpainting the static occluded background. The first challenge is addressed by the use of a convolutional network that learns a multi-class semantic segmentation of the image. The second challenge is approached with a generative adversarial model that, taking as input the original dynamic image and the computed dynamic/static binary mask, is capable of generating the final static image. This framework makes use of two new losses, one based on image steganalysis techniques, useful to improve the inpainting quality, and another one based on ORB features, designed to enhance feature matching between real and hallucinated image regions. To validate our approach, we perform an extensive evaluation on different tasks that are affected by dynamic entities, i.e., visual odometry, place recognition and multi-view stereo, with the hallucinated images. Code has been made available on https://github.com/bertabescos/EmptyCities_SLAM.

A survey: which features are required for dynamic visual simultaneous localization and mapping?

“…In contrast to applying clustering algorithms to low-level features, high-level features facilitate the clustering of map points belonging to independent objects with different dynamics, as well as the potential for detecting dynamic objects in one shot [ 100 ]. For rigid objects, features with the same semantic label always have the same motion label.…”

VIODE: A Simulated Dataset to Address the Challenges of Visual-Inertial Odometry in Dynamic Environments

“…The authors of Mask-SLAM and DynaSLAM evaluate proposal methods on their original dataset recorded in dynamic environments. Empty Cities [29] integrates dynamic object detection with a generative adversarial model to inpaint the dynamic objects and generate static scenes from images in dynamic environments.…”

Multi-Modal Visual Place Recognition in Dynamics-Invariant Perception Space

“…In this work, we share a similar line of thought with Berta et al [11], [14], but move one step forward to build a multi-modal dynamics-invariant perception space to improve feature matching in dynamic environments. This space is built by first designing a novel deep neural network architecture to reconstruct the static semantics (i.e., static semantic segmentation map) and static images from the dynamic images in a sequential manner.…”

“…The most related work is by Berta et al [11], who improve Pix2Pix [12] by performing conditioning on both the dynamic image and its dynamic mask under cGAN [13], to recover realistic static images. Recently, Berta et al [14] implement two more losses based on image steganalysis techniques and ORB features, respectively, to better recover reliable features.…”

“…On one hand, simply discarding dynamic contents as done in [8], [9] reduces the amount of available features and may cause failures in feature matching when the dynamic portions of the image tends to dominate the whole image in feature space. On the other hand, although those dynamicto-static image translation approaches [11], [14] are capable of generating visually realistic images, they easily introduce blur and artifacts, especially in areas associated with moving objects. Extracting features on such recovered static images will degrade feature matching to some extent.…”