Visual metrics for the evaluation of sensor data quality in outdoor perception

Brunner, Christopher; Peynot, Thierry

doi:10.1145/2377576.2377578

Cited by 10 publications

(18 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other works studying visual odometry performance in adverse conditions show correlations between image quantities, e.g. brightness, sharpness, blur, with the number of correspondences [10], [11]. Unlike these methods, our approach is theoretical in nature.…”

Section: B Heuristic Evaluationmentioning

confidence: 96%

“…These external sources of feedback which are not available during normal operation are known as "ground truth". There exist heuristic approaches for certain systems which do not rely on ground truth, for instance visual odometry [10], [11], but these are difficult to extend. We desire a way to evaluate perception performance that is generalizable and does not rely on ground-truth.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Introspective Evaluation of Perception Performance for Parameter Tuning without Ground Truth

Hu¹,

Kantor²

2017

Robotics: Science and Systems XIII

View full text Add to dashboard Cite

Abstract-Modern perception systems are notoriously complex, featuring dozens of interacting parameters that must be tuned to achieve good performance. Conventional tuning approaches require expensive ground truth, while heuristic methods are difficult to generalize. In this work, we propose an introspective ground-truth-free approach to evaluating the performance of a generic perception system. By using the posterior distribution estimate generated by a Bayesian estimator, we show that the expected performance can be estimated efficiently and without ground truth. Our simulated and physical experiments in a demonstrative indoor ground robot state estimation application show that our approach can order parameters similarly to using a ground-truth system, and is able to accurately identify top-performing parameters in varying contexts. In contrast, baseline approaches that reason only about observation log-likelihood fail in the face of challenging perceptual phenomena.

show abstract

Section: B Heuristic Evaluationmentioning

confidence: 96%

Section: Introductionmentioning

confidence: 99%

Introspective Evaluation of Perception Performance for Parameter Tuning without Ground Truth

Hu¹,

Kantor²

2017

Robotics: Science and Systems XIII

View full text Add to dashboard Cite

show abstract

“…In the telecommunication literature, methods have been proposed to explicitly evaluate the quality of image data for human consumption (I.T.U., ; Wang and Bovik, ; Winkler, ). Previous work by Brunner et al (, ) proposed a study of many of these image‐quality metrics and their relevance to outdoor robotic perception specifically for visual and IR imagery. Many quality metrics are designed to capture the errors caused by compression and transmission and are typically tailored to the human vision system.…”

Section: Related Work: Resilient Perception In Low‐visibility Conditionsmentioning

confidence: 99%

“…SE is defined as the entropy of distribution of intensities in a Sobel‐filtered image. This was shown to be a good indicator of image quality, especially in low‐visibility conditions (Brunner et al, ), and to be appropriate to evaluate both visual and infrared data.…”

Section: Related Work: Resilient Perception In Low‐visibility Conditionsmentioning

confidence: 99%

See 1 more Smart Citation

Selective Combination of Visual and Thermal Imaging for Resilient Localization in Adverse Conditions: Day and Night, Smoke and Fire

Brunner

Peynot

Vidal-Calleja

et al. 2013

Journal of Field Robotics

Self Cite

View full text Add to dashboard Cite

Long-term autonomy in robotics requires perception systems that are resilient to unusual but realistic conditions that will eventually occur during extended missions. For example, unmanned ground vehicles (UGVs) need to be capable of operating safely in adverse and low-visibility conditions, such as at night or in the presence of smoke. The key to a resilient UGV perception system lies in the use of multiple sensor modalities, e.g., operating at different frequencies of the electromagnetic spectrum, to compensate for the limitations of a single sensor type. In this paper, visual and infrared imaging are combined in a Visual-SLAM algorithm to achieve localization. We propose to evaluate the quality of data provided by each sensor modality prior to data combination. This evaluation is used to discard low-quality data, i.e., data most likely to induce large localization errors. In this way, perceptual failures are anticipated and mitigated. An extensive experimental evaluation is conducted on data sets collected with a UGV in a range of environments and adverse conditions, including the presence of smoke (obstructing the visual camera), fire, extreme heat (saturating the infrared camera), low-light conditions (dusk), and at night with sudden variations of artificial light. A total of 240 trajectory estimates are obtained using five different variations of data sources and data combination strategies in the localization method. In particular, the proposed approach for selective data combination is compared to methods using a single sensor type or combining both modalities without preselection. We show that the proposed framework allows for camera-based localization resilient to a large range of low-visibility conditions. C

show abstract

Perception Quality Evaluation with Visual and Infrared Cameras in Challenging Environmental Conditions

Brunner

Peynot

2014

Springer Tracts in Advanced Robotics

Self Cite

View full text Add to dashboard Cite

This work aims to contribute to the reliability and integrity of perceptual systems of unmanned ground vehicles (UGV). A method is proposed to evaluate the quality of sensor data prior to its use in a perception system by utilising a quality metric applied to heterogeneous sensor data such as visual and infrared camera images. The concept is illustrated specifically with sensor data that is evaluated prior to the use of the data in a standard SIFT feature extraction and matching technique. The method is then evaluated using various experimental data sets that were collected from a UGV in challenging environmental conditions, represented by the presence of airborne dust and smoke. In the first series of experiments, a motionless vehicle is observing a 'reference' scene, then the method is extended to the case of a moving vehicle by compensating for its motion. This paper shows that it is possible to anticipate degradation of a perception algorithm by evaluating the input data prior to any actual execution of the algorithm.

show abstract

Visual metrics for the evaluation of sensor data quality in outdoor perception

Cited by 10 publications

References 20 publications

Introspective Evaluation of Perception Performance for Parameter Tuning without Ground Truth

Introspective Evaluation of Perception Performance for Parameter Tuning without Ground Truth

Selective Combination of Visual and Thermal Imaging for Resilient Localization in Adverse Conditions: Day and Night, Smoke and Fire

Perception Quality Evaluation with Visual and Infrared Cameras in Challenging Environmental Conditions

Contact Info

Product

Resources

About