PerfLoc (Part 1): An extensive data repository for development of smartphone indoor localization apps

In recent years, indoor localization systems have been the object of significant research activity and of growing interest for their great expected social impact and their impressive business potential. Application areas include tracking and navigation, activity monitoring, personalized advertising, Active and Assisted Living (AAL), traceability, Internet of Things (IoT) networks, and Home-land Security. In spite of the numerous research advances and the great industrial interest, no canned solutions have yet been defined. The diversity and heterogeneity of applications, scenarios, sensor and user requirements, make it difficult to create uniform solutions. From that diverse reality, a main problem is derived that consists in the lack of a consensus both in terms of the metrics and the procedures used to measure the performance of the different indoor localization and navigation proposals. This paper introduces the general lines of the EvAAL benchmarking framework, which is aimed at a fair comparison of indoor positioning systems through a challenging competition under complex, realistic conditions. To evaluate the framework capabilities, we show how it was used in the 2016 Indoor Positioning and Indoor Navigation (IPIN) Competition. The 2016 IPIN competition considered three different scenario dimensions, with a variety of use cases: (1) pedestrian versus robotic navigation, (2) smartphones versus custom hardware usage and (3) real-time positioning versus off-line post-processing. A total of four competition tracks were evaluated under the same EvAAL benchmark framework in order to validate its potential to become a standard for evaluating indoor localization solutions. The experience gained during the competition and feedback from track organizers and competitors showed that the EvAAL framework is flexible enough to successfully fit the very different tracks and appears adequate to compare indoor positioning systems.

Section: Diversity Problem Reviewmentioning

confidence: 99%

Comparing the Performance of Indoor Localization Systems through the EvAAL Framework

Potortì

Park

Ruiz

et al. 2017

“…As the existing on-site and off-site indoor localization competitions, we select the following three competitions series as popular and dominant indoor localization competitions. PerLoc Competition, which was organized by National Institute of Standards and Technology (NIST) [ 3 , 4 ]; EvAAL (Evaluating AAL Systems through Competitive Benchmarking)/IPIN Competitions [ 5 , 6 , 7 ]; Indoor Localization Competitions, organized by Microsoft [ 8 ]. …”

Section: Survey Of the Existing Indoor Localization Competitions Amentioning

confidence: 99%

“…PerLoc Competition, which was organized by National Institute of Standards and Technology (NIST) [ 3 , 4 ];…”

Section: Survey Of the Existing Indoor Localization Competitions Amentioning

confidence: 99%

See 1 more Smart Citation

Off-Site Indoor Localization Competitions Based on Measured Data in a Warehouse

Ichikari

Kaji

Shimomura

et al. 2019

The performance of indoor localization methods is highly dependent on the situations in which they are used. Various competitions on indoor localization have been held for fairly comparing the existing indoor localization methods in shared and controlled testing environments. However, it is difficult to evaluate the practical performance in industrial scenarios through the existing competitions. This paper introduces two indoor localization competitions, which are named the “PDR Challenge in Warehouse Picking 2017” and “xDR Challenge for Warehouse Operations 2018” for tracking workers and vehicles in a warehouse scenario. For the PDR Challenge in Warehouse Picking 2017, we conducted a unique competition based on the data measured during the actual picking operation in an actual warehouse. We term the dead-reckoning of a vehicle as vehicle dead-reckoning (VDR), and the term “xDR” is derived from pedestrian dead-reckoning (PDR) plus VDR. As a sequel competition of the PDR Challenge in Warehouse Picking 2017, the xDR Challenge for Warehouse Operations 2018 was conducted as the world’s first competition that deals with tracking forklifts by VDR with smartphones. In the paper, first, we briefly summarize the existing competitions, and clarify the characteristics of our competitions by comparing them with other competitions. Our competitions have the unique capability of evaluating the practical performance in a warehouse by using the actual measured data as the test data and applying multi-faceted evaluation metrics. As a result, we successfully organize the competitions due to the many participants from many countries. As a conclusion of the paper, we summarize the findings of the competitions.

“…An inspiration for this approach is the Machine Learning Repository of the University of California Irvine (UCI) [9] and Kaggle [10], both created for evaluating machine learning algorithms with common databases. Although some related databases [11,12,13,14,15,16,17,18,19,20,21,22], interesting initiatives for benchmarking [23] and even competitions (Microsoft-IPSN [24,25,26,27], EvAAL [28,29,30,31,32] and EVARILOS [33,34]) for better evaluation of IPS already exist, no one encompasses the wide diversity in technologies, contexts and scenarios currently found in this field.…”

Section: Introductionmentioning

confidence: 99%

The Smartphone-Based Offline Indoor Location Competition at IPIN 2016: Analysis and Future Work

Torres-Sospedra

Jiménez

Knauth

et al. 2017

This paper presents the analysis and discussion of the off-site localization competition track, which took place during the Seventh International Conference on Indoor Positioning and Indoor Navigation (IPIN 2016). Five international teams proposed different strategies for smartphone-based indoor positioning using the same reference data. The competitors were provided with several smartphone-collected signal datasets, some of which were used for training (known trajectories), and others for evaluating (unknown trajectories). The competition permits a coherent evaluation method of the competitors’ estimations, where inside information to fine-tune their systems is not offered, and thus provides, in our opinion, a good starting point to introduce a fair comparison between the smartphone-based systems found in the literature. The methodology, experience, feedback from competitors and future working lines are described.