Requirements-Driven Test Generation for Autonomous Vehicles With Machine Learning Components

Tuncali, Cumhur Erkan; Fainekos, Georgios; Prokhorov, Danil; Ito, Hideto; Kapinski, James

doi:10.1109/tiv.2019.2955903

Cited by 91 publications

(64 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, test generation tools have started targeting autonomous systems under a simulation-based semantic testing framework similar to ours. In most of these works, visual scenarios are either fixed by hand [1,2,10,22,27,29,66,67], or are constrained due to the model or coverage criteria [3,45,50]. These analyses are shown to be preferable to the application of random noise on the input vector.…”

Section: Related Workmentioning

confidence: 99%

Paracosm: A Test Framework for Autonomous Driving Simulations

Majumdar

Mathur

Pirron

et al. 2021

Fundamental Approaches to Software Engineering

View full text Add to dashboard Cite

Systematic testing of autonomous vehicles operating in complex real-world scenarios is a difficult and expensive problem. We present Paracosm, a framework for writing systematic test scenarios for autonomous driving simulations. Paracosm allows users to programmatically describe complex driving situations with specific features, e.g., road layouts and environmental conditions, as well as reactive temporal behaviors of other cars and pedestrians. A systematic exploration of the state space, both for visual features and for reactive interactions with the environment is made possible. We define a notion of test coverage for parameter configurations based on combinatorial testing and low dispersion sequences. Using fuzzing on parameter configurations, our automatic test generator can maximize coverage of various behaviors and find problematic cases. Through empirical evaluations, we demonstrate the capabilities of Paracosm in programmatically modeling parameterized test environments, and in finding problematic scenarios.

show abstract

Section: Related Workmentioning

confidence: 99%

Paracosm: A Test Framework for Autonomous Driving Simulations

Majumdar

Mathur

Pirron

et al. 2021

Fundamental Approaches to Software Engineering

View full text Add to dashboard Cite

show abstract

“…Autonomous driving simulators are often used to test autonomous vehicles for the sake of efficiency and safety [30]- [33]. After testing popular autonomous simulators [34]- [37], we chose to run our experiments on the CARLA [8] (CAR Learning to Act) autonomous vehicle simulator, due to its feature-set and ease of source code modification .…”

Section: A Autonomous Vehicle Simulatormentioning

confidence: 99%

Attacking vision-based perception in end-to-end autonomous driving models

Boloor

Garimella

et al. 2020

Journal of Systems Architecture

View full text Add to dashboard Cite

Recent advances in machine learning, especially techniques such as deep neural networks, are enabling a range of emerging applications. One such example is autonomous driving, which often relies on deep learning for perception. However, deep learning-based perception has been shown to be vulnerable to a host of subtle adversarial manipulations of images. Nevertheless, the vast majority of such demonstrations focus on perception that is disembodied from end-to-end control. We present novel endto-end attacks on autonomous driving in simulation, using simple physically realizable attacks: the painting of black lines on the road. These attacks target deep neural network models for endto-end autonomous driving control. A systematic investigation shows that such attacks are easy to engineer, and we describe scenarios (e.g., right turns) in which they are highly effective. We define several objective functions that quantify the success of an attack and develop techniques based on Bayesian Optimization to efficiently traverse the search space of higher dimensional attacks. Additionally, we define a novel class of hijacking attacks, where painted lines on the road cause the driverless car to follow a target path. Through the use of network deconvolution, we provide insights into the successful attacks, which appear to work by mimicking activations of entirely different scenarios. Our code is available on https://github.com/xz-group/AdverseDrive Index Terms-machine learning, adversarial examples, autonomous driving, end-to-end learning, bayesian optimization

show abstract

“…We can see how the operational/practical constraints will be derived from many of the existing requirements on cruise control, lane following, etc, while we will see refined requirements for the sensing/monitoring capabilities within the vehicle. Verifying all of these aspects often involves a suite of (acceptable) testing schemes, such as requirements-driven test-case generation [240]. Just as the agent controlling an autonomous air vehicle can be verified with respect to the 'Rules of the Air' (see above), the agent controlling an autonomous road vehicle can be verified against the 'rules of the road' [6].…”

Section: Suitable Verification Validation and Analysis Techniquesmentioning

confidence: 99%

Towards a framework for certification of reliable autonomous systems

Fisher

Mascardi

Rozier

et al. 2020

Auton Agent Multi-Agent Syst

View full text Add to dashboard Cite

A computational system is called autonomous if it is able to make its own decisions, or take its own actions, without human supervision or control. The capability and spread of such systems have reached the point where they are beginning to touch much of everyday life. However, regulators grapple with how to deal with autonomous systems, for example how could we certify an Unmanned Aerial System for autonomous use in civilian airspace? We here analyse what is needed in order to provide verified reliable behaviour of an autonomous system, analyse what can be done as the state-of-the-art in automated verification, and propose a roadmap towards developing regulatory guidelines, including articulating challenges to researchers, to engineers, and to regulators. Case studies in seven distinct domains illustrate the article.

show abstract

Requirements-Driven Test Generation for Autonomous Vehicles With Machine Learning Components

Cited by 91 publications

References 55 publications

Paracosm: A Test Framework for Autonomous Driving Simulations

Paracosm: A Test Framework for Autonomous Driving Simulations

Attacking vision-based perception in end-to-end autonomous driving models

Towards a framework for certification of reliable autonomous systems

Contact Info

Product

Resources

About