On measuring the accuracy of SLAM algorithms

Abstract

In this paper, we address the problem of creating an objective benchmark for evaluating SLAM approaches. We propose a framework for analyzing the results of a SLAM approach based on a metric for measuring the error of the corrected trajectory. This metric uses only relative relations between poses and does not rely on a global reference frame. This overcomes serious shortcomings of approaches using a global reference frame to compute the error. Our method furthermore allows us to compare SLAM approaches that use different estimation techniques or different sensor modalities since all computations are made based on the corrected trajectory of the robot.We provide sets of relative relations needed to compute our metric for an extensive set of datasets frequently used in the robotics community. The relations have been obtained by manually matching laser-range observations to avoid the errors caused by matching algorithms. Our benchmark framework allows the user to easily analyze and objectively compare different SLAM approaches.

A comparison of SLAM algorithms based on a graph of relations

“…In most cases, however, such information is not available. See [16] for an example using aerial imagery.…”

“…To give a visual impression of the corresponding environments, Figure 2 illustrates maps obtained by executing state-of-the-art SLAM algorithms. All datasets, the manually verified relations, and map images are available online [17].…”

“…The manual work, however, has to be done only once for a dataset and then allows other researchers to evaluate their methods with low effort. Together with this paper, we provide a web page that hosts such manually matched relations for existing log files [17]. We furthermore provide evaluations for the results of three different mapping techniques, namely scan-matching, SLAM using Rao-Blackwellized particle filter [10], and a maximum likelihood SLAM approach based on the graph formulation [11], [21].…”

Advancing whisker based navigation through the implementation of Bio-Inspired whisking strategies

“…The first metric, which we called the Experience Metric (ExM), is novel and was designed specifically for use with the RatSLAM algorithm while the second, Energy Metric (EM), was derived from [15] as is a more general metric for evaluating SLAM algorithms. The reasons for introducing the Experience Metric is revealed in the results and described further in the discussion section of this paper, here we simply describe the principles of the new metric and the existing Energy Metric.…”

Towards Autonomous Robotic In-Situ Assembly on Unstructured Construction Sites Using Monocular Vision

“…During this research, it was realized that increasing the level of autonomy for construction robots requires high accuracy localization of the robot: from 3-5 cm indoor positional accuracy for contactless construction tasks such as spray-painting, to 2-3 mm accuracy for more precise tasks demanding direct contact between manipulator and building components [11]. This requirement has posed a significant challenge for ARC because even by using current state-of-the-art simultaneous localization and mapping (SLAM) techniques, such accuracy is hard to achieve at large scales [12]. In order to address this issue, the authors chose to use computer-vision-based pose estimation algorithms that can achieve high accuracy locally around a visual marker [13,14].…”