Crashing Simulated Planes is Cheap: Can Simulation Detect Robotics Bugs Early?

Timperley, Christopher Steven; Afzal, Afsoon; Katz, Deborah; Hernandez, Jam Marcos; Goues, Claire Le

doi:10.1109/icst.2018.00040

Cited by 47 publications

(21 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Out of the 33 bugs extracted from the code commits, only one requires a high fidelity to be replicated in simulation (the bug is related to mechanical vibration that was not simulated). Timperley et al (2018) came to a similar conclusion for bugs in the open-source ArduPilot system: the majority of them surface under simple conditions that can be easily reproduced in software-based simulation.…”

Section: Fidelity Of the Simulationmentioning

confidence: 68%

“…The outcomes regarding test effectiveness may be more application-dependent. However, we consider it useful to study effectiveness for a real-scale industrial example, which supplements other empirical results on academic examples, e.g., Sotiropoulos et al (2017); Timperley et al (2018). Finally, the approaches adopted for generating tests and detecting misbehavior are general, their principle can be reused outside the case study.…”

Section: Threats To Validitymentioning

confidence: 99%

See 1 more Smart Citation

The virtual lands of Oz: testing an agribot in simulation

et al. 2020

View full text Add to dashboard Cite

HAL is a multidisciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

show abstract

Section: Fidelity Of the Simulationmentioning

confidence: 68%

Section: Threats To Validitymentioning

confidence: 99%

The virtual lands of Oz: testing an agribot in simulation

et al. 2020

View full text Add to dashboard Cite

show abstract

“…In [9], Procedural Content Generation (PCG)-an automated technique borrowed from the video games field [10]-is used to generate 2D environments to test the path-following algorithm with collision avoidance of a mobile robot, demonstrating that bugs can be found even with low physical fidelity. [11] came to the same conclusion, showing that the majority of the bugs of the ArduPilot System are exposed by trivial conditions, and do not require the reproduction of complex physical phenomena to be revealed. Previous work at LAAS reinforces this idea, as in the study on the navigation software of the mobile robot Mana [2], where out of 33 bugs, the only bug that needed high fidelity to be replicated was related to mechanical vibration, which was not included in the simulation.…”

Section: Related Workmentioning

confidence: 70%

Simulation-Based Testing to Improve Safety of Autonomous Robots

Sartori

2019

2019 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)

View full text Add to dashboard Cite

Autonomous systems are becoming increasingly popular among industries as well as end-users, and are deployed in numerous tasks. To improve their reliability and to avoid critical failures that impact safety, their testing aims at ensuring that their behavior and decisions are acceptable even in scenarios that have not been foreseen by the developers. As testing of these systems is usually done through field testing, which is costly and is limited in the reproducible scenarios, system-level pre-validation can be done in virtual worlds through simulation, to discover faults and fix them before deploying the system in the real world. However, there is no current standard procedure to conduct simulation-based testing and to ensure satisfying coverage of the most critical scenarios for the system under test (SUT). The aim of this experimental work is to improve and automate the steps of the simulation-based testing related to the generation and selection of test inputs, the exploitation of the results, and the incorporation of dynamic agents in the tests.

show abstract

“…Scenic automatically determines the spatial relationships between objects in the scene such that they conform to the specifications of the scenario and do not collide with each other. 2 Scenic arranges objects in the scene by treating each objects as a bounding rectangle on a two-dimensional plane. At the time of writing, Scenic is unable to arrange objects in three dimensions.…”

Section: Background a Scenicmentioning

confidence: 99%

GzScenic: Automatic Scene Generation for Gazebo Simulator

Afzal,

Goues,

Timperley

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Testing robotic and cyberphysical systems in simulation require specifications of the simulated environments (i.e., scenes). The Scenic domain-specific language provides a high-level probabilistic programming language that allows users to specify scenarios for simulation. Scenic automatically generates concrete scenes that can be rendered by simulators. However, Scenic is mainly designed for autonomous vehicle simulation and does not support the most popular generalpurpose simulator: Gazebo. In this work, we present GzScenic; a tool that automatically generates scenes for simulation in Gazebo. GzScenic automatically generates both the models required for running Scenic on the scenarios, and the models that Gazebo requires for running the simulation.

show abstract

Crashing Simulated Planes is Cheap: Can Simulation Detect Robotics Bugs Early?

Cited by 47 publications

References 30 publications

The virtual lands of Oz: testing an agribot in simulation

The virtual lands of Oz: testing an agribot in simulation

Simulation-Based Testing to Improve Safety of Autonomous Robots

GzScenic: Automatic Scene Generation for Gazebo Simulator

Contact Info

Product

Resources

About