Positioning evaluation and ground truth definition for real life use cases

Osa, Carlos Martinez de la; Anagnostopoulos, Grigorios G.; Togneri, Mauricio; Deriaz, Michel; Konstantas, Dimitri

doi:10.1109/ipin.2016.7743650

Cited by 11 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There has been no common framework in which the DAE estimates of positioning systems are consistently compared. An early reference to this potential is made by a framework of evaluating [28] and tuning [31,32] the positioning systems, proposed in 2016, where the evaluation of the DAE [28] is listed as one of the metrics that the proposed framework can evaluate and tune. Nevertheless, in the experimentation sections of this framework, only the evaluation of location estimates is exemplified.…”

Section: Evaluation Metrics and Methodologiesmentioning

confidence: 99%

Can I Trust This Location Estimate? Reproducibly Benchmarking the Methods of Dynamic Accuracy Estimation of Localization

Anagnostopoulos

Kalousis

2022

Sensors

Self Cite

View full text Add to dashboard Cite

Despite the great attention that the research community has paid to the creation of novel indoor positioning methods, a rather limited volume of works has focused on the confidence that Indoor Positioning Systems (IPS) assign to the position estimates that they produce. The concept of estimating, dynamically, the accuracy of the position estimates provided by an IPS has been sporadically studied in the literature of the field. Recently, this concept has started being studied as well in the context of outdoor positioning systems of Internet of Things (IoT) based on Low-Power Wide-Area Networks (LPWANs). What is problematic is that the consistent comparison of the proposed methods is quasi nonexistent: new methods rarely use previous ones as baselines; often, a small number of evaluation metrics are reported while different metrics are reported among different relevant publications, the use of open data is rare, and the publication of open code is absent. In this work, we present an open-source, reproducible benchmarking framework for evaluating and consistently comparing various methods of Dynamic Accuracy Estimation (DAE). This work reviews the relevant literature, presenting in a consistent terminology commonalities and differences and discussing baselines and evaluation metrics. Moreover, it evaluates multiple methods of DAE using open data, open code, and a rich set of relevant evaluation metrics. This is the first work aiming to establish the state of the art of methods of DAE determination in IPS and in LPWAN positioning systems, through an open, transparent, holistic, reproducible, and consistent evaluation of the methods proposed in the relevant literature.

show abstract

Section: Evaluation Metrics and Methodologiesmentioning

confidence: 99%

Can I Trust This Location Estimate? Reproducibly Benchmarking the Methods of Dynamic Accuracy Estimation of Localization

Anagnostopoulos

Kalousis

2022

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…Liu et al [15] proposes a novel way to compute similarity between moving objects, based on geographic and semantic components of the movements. de la Osa et al [16] deals with the lack of a predominant solution for defining the ground truth when comparing indoor position estimates. Anagnostopoulos et al [17] defines a novel dynamic evaluation procedure with predefined geometrical paths (see [18]) to capture real-life usage of the scenarios, as well as other features such as ease of deployment and cost-efficiency.…”

Section: Positioning Error In Indoor Positioning Systemsmentioning

confidence: 99%

Beyond Euclidean Distance for Error Measurement in Pedestrian Indoor Location

Mendoza-Silva

Torres-Sospedra²,

Potortì

et al. 2021

IEEE Trans. Instrum. Meas.

View full text Add to dashboard Cite

Indoor Positioning Systems suffer from a lack of standard evaluation procedures enabling credible comparisons: this is one of the main challenges hindering their widespread market adoption. Traditionally, accuracy evaluation is based on positioning errors defined as the Euclidean distance between the true positions and the estimated positions. While Euclidean is simple, it ignores obstacles and floor transitions. In this paper, we describe procedures that measure a positioning error defined as the length of the pedestrian path that connects the estimated position to the true position. The procedures apply pathfinding on floor maps using Visibility Graphs or Navigational Meshes for vector maps, and Fast Marching for raster maps. Multifloor and multi-building paths use information on vertical inbuilding communication ways and outdoor paths. The proposed measurement procedures are applied to position estimates provided by the Indoor Positioning Systems that participated in the EvAAL-ETRI 2015 competition. Procedures are compared in terms of pedestrian path realism, indoor model complexity, path computation time and error magnitudes. The Visibility Graphs algorithm computes shortest distance paths; Navigational Meshes produces very similar paths with significantly shorter computation time; Fast Marching computes longer, more naturallooking paths at the expense of longer computation time and memory size. The 75 th percentile of the measured error differs among the methods from 2.2 m to 3.7 m across the evaluation sets.

show abstract

“…This concept of error prediction is far from constituting a usually advertised feature of indoor positioning systems and relevant publications, or a subject of comparison in the competitions of the field [9] which mainly compare the accuracy of the location predictions. Nevertheless, a framework of evaluating [10] and tuning [11], [12] positioning systems, proposed in 2016, lists the DAE as one of the metrics that the framework can evaluate and tune. Different approaches have been proposed regarding the way in which the certainty over a produced location estimate can be estimated.…”

Section: Related Workmentioning

confidence: 99%

Analysing the Data-Driven Approach of Dynamically Estimating Positioning Accuracy

Anagnostopoulos

Kalousis

2021

ICC 2021 - IEEE International Conference on Communications

Self Cite

View full text Add to dashboard Cite

The primary expectation from positioning systems is for them to provide the users with reliable estimates of their position. An additional piece of information that can greatly help the users utilize position estimates is the level of uncertainty that a positioning system assigns to the position estimate it produced. The concept of dynamically estimating the accuracy of position estimates of fingerprinting positioning systems has been sporadically discussed over the last decade in the literature of the field, where mainly handcrafted rules based on domain knowledge have been proposed. The emergence of IoT devices and the proliferation of data from Low Power Wide Area Networks (LPWANs) have facilitated the conceptualization of data-driven methods of determining the estimated certainty over position estimates. In this work, we analyze the data-driven approach of determining the Dynamic Accuracy Estimation (DAE), considering it in the broader context of a positioning system. More specifically, with the use of a public LoRaWAN dataset, the current work analyses: the repartition of the available training set between the tasks of determining the location estimates and the DAE, the concept of selecting a subset of the most reliable estimates, and the impact that the spatial distribution of the data has to the accuracy of the DAE. The work provides a wide overview of the data-driven approach of DAE determination in the context of the overall design of a positioning system.

show abstract

Positioning evaluation and ground truth definition for real life use cases

Cited by 11 publications

References 12 publications

Can I Trust This Location Estimate? Reproducibly Benchmarking the Methods of Dynamic Accuracy Estimation of Localization

Can I Trust This Location Estimate? Reproducibly Benchmarking the Methods of Dynamic Accuracy Estimation of Localization

Beyond Euclidean Distance for Error Measurement in Pedestrian Indoor Location

Analysing the Data-Driven Approach of Dynamically Estimating Positioning Accuracy

Contact Info

Product

Resources

About