Testing of Advanced Driver Assistance Towards Automated Driving: A Survey and Taxonomy on Existing Approaches and Open Questions

Stellet, Jan Erik; Zofka, Marc René; Schumacher, Jan; Schamm, Thomas; Niewels, Frank; Zöllner, J. Marius

doi:10.1109/itsc.2015.236

Cited by 110 publications

(91 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent research has proposed methods aimed at reducing the amount of realworld testing required from manufacturers. Such methods include testing in virtual and hardware-in-the-loop simulations, limiting the scope of real-world testing to safety-critical scenarios, and using threat measures from near-collisions to quantify system safety [1], [30], [35]- [37].…”

Section: Application To Autonomous Vehiclesmentioning

confidence: 99%

Closed-Loop Policies for Operational Tests of Safety-Critical Systems

Morton

Wheeler

Kochenderfer

2018

IEEE Trans. Intell. Veh.

View full text Add to dashboard Cite

Manufacturers of safety-critical systems must make the case that their product is sufficiently safe for public deployment. Much of this case often relies upon critical event outcomes from real-world testing, requiring manufacturers to be strategic about how they allocate testing resources in order to maximize their chances of demonstrating system safety. This work frames the partially observable and belief-dependent problem of test scheduling as a Markov decision process, which can be solved efficiently to yield closed-loop manufacturer testing policies. By solving for policies over a wide range of problem formulations, we are able to provide high-level guidance for manufacturers and regulators on issues relating to the testing of safety-critical systems. This guidance spans an array of topics, including circumstances under which manufacturers should continue testing despite observed incidents, when manufacturers should test aggressively, and when regulators should increase or reduce the real-world testing requirements for safety-critical systems.

show abstract

Section: Application To Autonomous Vehiclesmentioning

confidence: 99%

Closed-Loop Policies for Operational Tests of Safety-Critical Systems

Morton

Wheeler

Kochenderfer

2018

IEEE Trans. Intell. Veh.

View full text Add to dashboard Cite

show abstract

“…The amount of collected field data from driving studies is increasing rapidly and these data are extensively used for the research, development, assessment, and evaluation of driving-related topics; for example, see Klauer et al (2006), Williamson et al (2011), Broggi et al (2013), Sadigh et al (2014), Zofka et al (2015), Dingus et al (2016), de Gelder and Paardekooper (2017), P€ utz et al (2017), Elrofai et al (2018), Krajewski et al (2018), and Ploeg et al (2018). For any work that depends on data, it is important to know how complete the data are.…”

Section: Introductionmentioning

confidence: 99%

“…For any work that depends on data, it is important to know how complete the data are. As mentioned by various authors (Alvarez et al 2017;Geyer et al 2014;Stellet et al 2015), especially when deducing safety claims based on collected data; for example, through testing scenarios based on collected data, we require knowledge about the degree of completeness of the data set used. Hence, questions like "Do we have enough data?"…”

Section: Introductionmentioning

confidence: 99%

Safety assessment of automated vehicles: how to determine whether we have collected enough field data?

Gelder

Paardekooper

Camp³

et al. 2019

Traffic Injury Prevention

View full text Add to dashboard Cite

Objective: The amount of collected field data from naturalistic driving studies is quickly increasing. The data are used for, among others, developing automated driving technologies (such as crash avoidance systems), studying driver interaction with such technologies, and gaining insights into the variety of scenarios in real-world traffic. Because data collection is time consuming and requires high investments and resources, questions like "Do we have enough data?," "How much more information can we gain when obtaining more data?," and "How far are we from obtaining completeness?" are highly relevant. In fact, deducing safety claims based on collected data-for example, through testing scenarios based on collected data-requires knowledge about the degree of completeness of the data used. We propose a method for quantifying the completeness of the so-called activities in a data set. This enables us to partly answer the aforementioned questions. Method: In this article, the (traffic) data are interpreted as a sequence of different so-called scenarios that can be grouped into a finite set of scenario classes. The building blocks of scenarios are the activities. For every activity, there exists a parameterization that encodes all information in the data of each recorded activity. For each type of activity, we estimate a probability density function (pdf) of the associated parameters. Our proposed method quantifies the degree of completeness of a data set using the estimated pdfs. Results: To illustrate the proposed method, 2 different case studies are presented. First, a case study with an artificial data set, of which the underlying pdfs are known, is carried out to illustrate that the proposed method correctly quantifies the completeness of the activities. Next, a case study with real-world data is performed to quantify the degree of completeness of the acquired data for which the true pdfs are unknown. Conclusion: The presented case studies illustrate that the proposed method is able to quantify the degree of completeness of a small set of field data and can be used to deduce whether sufficient data have been collected for the purpose of the field study. Future work will focus on applying the proposed method to larger data sets. The proposed method will be used to evaluate the level of completeness of the data collection on Singaporean roads, aimed at defining relevant test cases for the autonomous vehicle road approval procedure that is being developed in Singapore. ARTICLE HISTORY

show abstract

“…The ultimate goal of autonomous road vehicles is to provide a safe and reliable transportation. But safety is a subjective concept that is hard to assess numerically and systematically [1]. One way to assess it is through events that demonstrates the lack of safety like the rate of crashes, injuries and deaths per traveled distance.…”

Section: Introductionmentioning

confidence: 99%

Verisimilar Percept Sequences Tests for Autonomous Driving Intelligent Agent Assessment

Watanabe

Wolf

2018

2018 Latin American Robotic Symposium, 2018 Brazilian Symposium on Robotics (SBR) and 2018 Workshop on Robotics in Education (W

View full text Add to dashboard Cite

The autonomous car technology promises to replace human drivers with safer driving systems. But although autonomous cars can become safer than human drivers this is a long process that is going to be refined over time. Before these vehicles are deployed on urban roads a minimum safety level must be assured. Since the autonomous car technology is still under development there is no standard methodology to evaluate such systems. It is important to completely understand the technology that is being developed to design efficient means to evaluate it. In this paper we assume safety-critical systems reliability as a safety measure. We model an autonomous road vehicle as an intelligent agent and we approach its evaluation from an artificial intelligence perspective. Our focus is the evaluation of perception and decision making systems and also to propose a systematic method to evaluate their integration in the vehicle. We identify critical aspects of the data dependency from the artificial intelligence state of the art models and we also propose procedures to reproduce them.

show abstract

Testing of Advanced Driver Assistance Towards Automated Driving: A Survey and Taxonomy on Existing Approaches and Open Questions

Cited by 110 publications

References 50 publications

Closed-Loop Policies for Operational Tests of Safety-Critical Systems

Closed-Loop Policies for Operational Tests of Safety-Critical Systems

Safety assessment of automated vehicles: how to determine whether we have collected enough field data?

Verisimilar Percept Sequences Tests for Autonomous Driving Intelligent Agent Assessment

Contact Info

Product

Resources

About