Characterizing Visual Localization and Mapping Datasets

Abstract

Benchmarking mapping and motion estimation algorithms is established practice in robotics and computer vision. As the diversity of datasets increases, in terms of the trajectories, models, and scenes, it becomes a challenge to select datasets for a given benchmarking purpose. Inspired by the Wasserstein distance, this paper addresses this concern by developing novel metrics to evaluate trajectories and the environments without relying on any SLAM or motion estimation algorithm. The metrics, which so far have been missing in the research community, can be applied to the plethora of datasets that exist. Additionally, to improve the robotics SLAM benchmarking, the paper presents a new dataset for visual localization and mapping algorithms. A broad range of real-world trajectories is used in very high-quality scenes and a rendering framework to create a set of synthetic datasets with groundtruth trajectory and dense map which are representative of key SLAM applications such as virtual reality (VR), micro aerial vehicle (MAV) flight, and ground robotics.

SLAMBench 3.0: Systematic Automated Reproducible Evaluation of SLAM Systems for Robot Vision Challenges and Scene Understanding

“…In total, SLAMBench 3.0 contains 6 new metrics, 4 new datasets and 5 new algorithms. In future, we plan to include trajectory difficulty metrics such as the metrics described in [54] and [55], and also to extend this paper to include systems that use multiple cameras [56].…”

Loop closure detection using local 3D deep descriptors

“…Firstly, we compare our approach against recent LiDAR-based loop closure approaches, namely Over-lapNet [10] and LiDAR Iris [8], using the LiDAR dataset KITTI odometry [40]. Secondly, we embed our loop closure method in a recent visual SLAM system, and test it on a real RGBD dataset, TUM-RGBD [20], and on a synthetic RGBD dataset, ICL [21]. We choose the edge-based visual SLAM method, namely RESLAM [2], as the point clouds produced by the mapping are different from those of KITTI.…”

“…Moreover, we show the efficacy of our approach to RGBD SLAM systems by embedding it into RESLAM [2], a recent RGBD edge-based SLAM approach. We evaluate the absolute trajectory error [20] on the real-world TUM-RGBD [20] and synthetic ICL [21] datasets. Our approach enables RESLAM to detect loops more frequently and to estimate 6DoF transformations more accurately than those estimated with the RESLAMS's original loop closure method.…”

A large-scale dataset for indoor visual localization with high-precision ground truth

“…In addition, the query and database images are captured at different places and the field of view (FoV) overlaps between them are small. The 7 Scenes dataset (Shotton et al, 2013) and ICL dataset (Saeedi et al, 2019) focus on indoor simultaneous localization and mapping (SLAM) problems, which provides a very different dataset to ours. They record small-scale trajectories which are mostly less than 10 m covering less than 100 m 2 area.…”