Evaluating Perception Systems for Autonomous Vehicles Using Quality Temporal Logic

Dokhanchi, Adel; Amor, Heni Ben; Deshmukh, Jyotirmoy V.; Fainekos, Georgios

doi:10.1007/978-3-030-03769-7_23

Cited by 38 publications

(41 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A formal language for specifying requirements on the performance of object detection in the absence of reference data is proposed in Refs. [120,121].…”

Section: Concrete Approaches Of Specifying Perceptionmentioning

confidence: 99%

A Review of Testing Object-Based Environment Perception for Safe Automated Driving

2022

View full text Add to dashboard Cite

Safety assurance of automated driving systems must consider uncertain environment perception. This paper reviews literature addressing how perception testing is realized as part of safety assurance. The paper focuses on testing for verification and validation purposes at the interface between perception and planning, and structures the analysis along the three axes (1) test criteria and metrics, (2) test scenarios, and (3) reference data. Furthermore, the analyzed literature includes related safety standards, safety-independent perception algorithm benchmarking, and sensor modeling. It is found that the realization of safety-oriented perception testing remains an open issue since challenges concerning the three testing axes and their interdependencies currently do not appear to be sufficiently solved.

show abstract

“…A formal language for specifying requirements on the performance of object detection in the absence of reference data is proposed in Refs. [120,121].…”

Section: Concrete Approaches Of Specifying Perceptionmentioning

confidence: 99%

A Review of Testing Object-Based Environment Perception for Safe Automated Driving

2022

View full text Add to dashboard Cite

show abstract

“…While there is work on evaluating performance of perception with temporal logic, those formal specifications are defined over image data streams, and must be manually formalized for each scenario / data stream [15], [16]. Often, there is high variability in the performance of perception models in seemingly similar environments, such as variations in sun angle [17].…”

Section: Introductionmentioning

confidence: 99%

Leveraging Classification Metrics for Quantitative System-Level Analysis with Temporal Logic Specifications

Badithela

Wongpiromsarn

Murray

2021

2021 60th IEEE Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

“…Timed Quality Temporal Logic (TQTL) [5], and Spatio-temporal Quality Logic (STQL) [14] are extensions to MTL that incorporate the semantics for reasoning about data from perception systems specifically. In STQL, which is in itself an extension of TQTL, the syntax defines operators to reason about discrete IDs and classes of objects, along with set operations on the spatial artifacts, like bounding boxes, outputted by perception systems.…”

Section: Introductionmentioning

confidence: 99%

“…1. We show how TQTL [5] and STQL [14] can be used to express correctness properties for perception algorithms. 2.…”

Section: Introductionmentioning

confidence: 99%

PerceMon: Online Monitoring for Perception Systems

Balakrishnan¹,

Deshmukh²,

Hoxha³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Perception algorithms in autonomous vehicles are vital for the vehicle to understand the semantics of its surroundings, including detection and tracking of objects in the environment. The outputs of these algorithms are in turn used for decision-making in safety-critical scenarios like collision avoidance, and automated emergency braking. Thus, it is crucial to monitor such perception systems at runtime. However, due to the high-level, complex representations of the outputs of perception systems, it is a challenge to test and verify these systems, especially at runtime. In this paper, we present a runtime monitoring tool, PerceMon that can monitor arbitrary specifications in Timed Quality Temporal Logic (TQTL) and its extensions with spatial operators. We integrate the tool with the CARLA autonomous vehicle simulation environment and the ROS middleware platform while monitoring properties on stateof-the-art object detection and tracking algorithms.

show abstract

Evaluating Perception Systems for Autonomous Vehicles Using Quality Temporal Logic

Cited by 38 publications

References 11 publications

A Review of Testing Object-Based Environment Perception for Safe Automated Driving

A Review of Testing Object-Based Environment Perception for Safe Automated Driving

Leveraging Classification Metrics for Quantitative System-Level Analysis with Temporal Logic Specifications

PerceMon: Online Monitoring for Perception Systems

Contact Info

Product

Resources

About