Hilti-Oxford Dataset: A Millimeter-Accurate Benchmark for Simultaneous Localization and Mapping

Abstract

Simultaneous Localization and Mapping (SLAM) is being deployed in real-world applications, however many stateof-the-art solutions still struggle in many common scenarios. A key necessity in progressing SLAM research is the availability of high-quality datasets and fair and transparent benchmarking. To this end, we have created the Hilti-Oxford Dataset, to push stateof-the-art SLAM systems to their limits. The dataset has a variety of challenges ranging from sparse and regular construction sites to a 17th century neoclassical building with fine details and curved surfaces. To encourage multi-modal SLAM approaches, we designed a data collection platform featuring a lidar, five cameras, and an IMU (Inertial Measurement Unit). With the goal of benchmarking SLAM algorithms for tasks where accuracy and robustness are paramount, we implemented a novel ground truth collection method that enables our dataset to accurately measure SLAM pose errors with millimeter accuracy. To further ensure accuracy, the extrinsics of our platform were verified with a micrometer-accurate scanner, and temporal calibration was managed online using hardware time synchronization. The multi-modality and diversity of our dataset attracted a large field of academic and industrial researchers to enter the second edition of the Hilti SLAM challenge, which concluded in June 2022. The results of the challenge show that while the top three teams could achieve an accuracy of 2 cm or better for some sequences, the performance dropped off in more difficult sequences.

A fully integrated system for hardware-accelerated TSDF SLAM with LiDAR sensors (HATSDF SLAM)

“…Furthermore, to evaluate the quality of the invented algorithms in realistic scenarios, new benchmark datasets are required that allow to evaluate many trajectories in challenging environments. To tackle this problem, we plan to provide a repository of reference TSDF maps with many different trajectories captured with a laser tracking system to provide a benchmarking environment for the development of such algorithms similar to the established KITTY [24] and Hilti [25] datasets for SLAM.…”

Large-Scale LiDAR Consistent Mapping Using Hierarchical LiDAR Bundle Adjustment

“…The parameters for our method are τ 1 = 0.0625, τ 2 = 3, 1 = 8 • and 2 = 10 • . We used the dataset HILTI 2022 [34] to evaluate precision and efficiency of these five methods. The accuracy of groundtruth in HILTI reaches millimeter scale, making it suitable for testing the performance of SLAM.…”

Quatro++: Robust global registration exploiting ground segmentation for loop closing in LiDAR SLAM

“…We conducted experiments with various datasets to evaluate the performance of Quatro++ (and Quatro). Accordingly, the following datasets were used: KITTI dataset (Geiger et al, 2012(Geiger et al, , 2013 to evaluate the success rate and robustness of global registration, NAVER LABS localization dataset (Lee et al, 2021) to check the applicability to more sparse LiDAR scans, MulRan dataset (Kim et al, 2020) to integrate our proposed method as a loop closing module, and Hilti-Oxford dataset (Zhang et al, 2022) to show feasibility of our Quatro++ on hand-held sensor configurations. These datasets were captured by Velodyne HDL-64E, Velodyne VLP-16, Ouster OS1-64, and HESAI XT32, respectively.…”