Automated Identification and Qualitative Characterization of Safety Concerns Reported in UAV Software Platforms

Sorbo, Andrea Di; Zampetti, Fiorella; Visaggio, Corrado Aaron; Penta, Massimiliano Di; Panichella, Sebastiano

doi:10.1145/3564821

Cited by 21 publications

(5 citation statements)

References 128 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The automated testing of cyber-physical systems (CPSs) remains an ongoing research challenge [60,74]. In this context, simulation-based testing emerges as a promising approach to enhance testing practices for safety-critical systems [10,11,13,48,53] and to support test automation [4,5,69,70,72].…”

Section: Simulation-based Testingmentioning

confidence: 99%

“…In this context, simulation-based testing emerges as a promising approach to enhance testing practices for safety-critical systems [10,11,13,48,53] and to support test automation [4,5,69,70,72]. Past research on testing CPS in simulation environments focused on monitoring CPS and predicting unsafe states [60,64] of the systems using simulation environments [64,73] as well as generating scenarios programmatically [51] or based on real-world observations [23,63]. Recent research also proposed cost-effective regression testing techniques, including test selection [10], prioritization [7,11] and minimization techniques to expose CPS faults or bugs earlier in the development and testing process.…”

Section: Simulation-based Testingmentioning

confidence: 99%

“…For instance, humans no longer rely on vacuuming their houses or mowing their grasses; nowadays, we have robots that can do (or will do) our chores [8]. However, specific safety-critical instances of autonomous systems such as unmanned aerial vehicles (UAVs) and self-driving cars (SDCs) [35,36,60,62,64,74] can experience failures that can harm humans or the environment [28].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?

Birchler,

Mohammed,

Rani

et al. 2024

Proc. ACM Softw. Eng.

Self Cite

View full text Add to dashboard Cite

Software metrics such as coverage or mutation scores have been investigated for the automated quality assessment of test suites. While traditional tools rely on software metrics, the field of self-driving cars (SDCs) has primarily focused on simulation-based test case generation using quality metrics such as the out-of-bound (OOB) parameter to determine if a test case fails or passes. However, it remains unclear to what extent this quality metric aligns with the human perception of the safety and realism of SDCs. To address this (reality) gap, we conducted an empirical study involving 50 participants to investigate the factors that determine how humans perceive SDC test cases as safe, unsafe, realistic, or unrealistic. To this aim, we developed a framework leveraging virtual reality (VR) technologies, called SDC-Alabaster, to immerse the study participants into the virtual environment of SDC simulators. Our findings indicate that the human assessment of safety and realism of failing/passing test cases can vary based on different factors, such as the test's complexity and the possibility of interacting with the SDC. Especially for the assessment of realism, the participants' age leads to a different perception. This study highlights the need for more research on simulation testing quality metrics and the importance of human perception in evaluating SDCs.

show abstract

Section: Simulation-based Testingmentioning

confidence: 99%

Section: Simulation-based Testingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?

Birchler,

Mohammed,

Rani

et al. 2024

Proc. ACM Softw. Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Building openly available corpora using the appropriate configuration for the generators, with a description of the characteristics applicable evaluations, would avoid such issues. More in general, having the JUGE infrastructure and its associated standards available to industrial and academic research communities can open the road for more systematic testing for other languages (e.g., Python) as well as the definition of testing pipelines to identify bugs and imperfections of systems in other application domains [86][87][88][89]. The availability of such technologies can also impact computer science education, with available tools that can be used in practical courses.…”

Section: Future Applicationsmentioning

confidence: 99%

JUGE: An infrastructure for benchmarking Java unit test generators

Devroey

Gambi

Galeotti

et al. 2022

Software Testing Verif & Rel

Self Cite

View full text Add to dashboard Cite

Summary Researchers and practitioners have designed and implemented various automated test case generators to support effective software testing. Such generators exist for various languages (e.g., Java, C#, or Python) and various platforms (e.g., desktop, web, or mobile applications). The generators exhibit varying effectiveness and efficiency, depending on the testing goals they aim to satisfy (e.g., unit‐testing of libraries versus system‐testing of entire applications) and the underlying techniques they implement. In this context, practitioners need to be able to compare different generators to identify the most suited one for their requirements, while researchers seek to identify future research directions. This can be achieved by systematically executing large‐scale evaluations of different generators. However, executing such empirical evaluations is not trivial and requires substantial effort to select appropriate benchmarks, setup the evaluation infrastructure, and collect and analyse the results. In this Software Note, we present our JUnit Generation Benchmarking Infrastructure (JUGE) supporting generators (search‐based, random‐based, symbolic execution, etc.) seeking to automate the production of unit tests for various purposes (validation, regression testing, fault localization, etc.). The primary goal is to reduce the overall benchmarking effort, ease the comparison of several generators, and enhance the knowledge transfer between academia and industry by standardizing the evaluation and comparison process. Since 2013, several editions of a unit testing tool competition, co‐located with the Search‐Based Software Testing Workshop, have taken place where JUGE was used and evolved. As a result, an increasing amount of tools (over 10) from academia and industry have been evaluated on JUGE, matured over the years, and allowed the identification of future research directions. Based on the experience gained from the competitions, we discuss the expected impact of JUGE in improving the knowledge transfer on tools and approaches for test generation between academia and industry. Indeed, the JUGE infrastructure demonstrated an implementation design that is flexible enough to enable the integration of additional unit test generation tools, which is practical for developers and allows researchers to experiment with new and advanced unit testing tools and approaches.

show abstract

“…To deal with such safety-related challenges, there is an increasing interest in adopting agile development paradigms within the CPS safety-critical domains [6,7] to identify hazards and elicit safety requirements iteratively [8]. Consequently, researchers proposed the usage of Digital-Twins 1 technologies to simulate and test CPSs in a diversified set of scenarios [9,10,11,12,13] to support testing automation [14,15], regression testing [12,16], and debugging [17,18] activities.…”

Section: Introductionmentioning

confidence: 99%

Cost-effective Simulation-based Test Selection in Self-driving Cars Software with SDC-Scissor

Birchler

Ganz

Khatiri

et al. 2022

2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)

View full text Add to dashboard Cite

Simulation platforms facilitate the continuous development of complex systems such as self-driving cars (SDCs). However, previous results on testing SDCs using simulations have shown that most of the automatically generated tests do not strongly contribute to establishing confidence in the quality and reliability of the SDC. Therefore, those tests can be characterized as "uninformative", and running them generally means wasting precious computational resources. We address this issue with SDC-Scissor, a framework that leverages Machine Learning to identify simulation-based tests that are unlikely to detect faults in the SDC software under test and skip them before their execution. Consequently, by filtering out those tests, SDC-Scissor reduces the number of long-running simulations to execute and drastically increases the cost-effectiveness of simulation-based testing of SDCs software. Our evaluation concerning two large datasets and around 12'000 tests showed that SDC-Scissor achieved a higher classification F1-score (between 47% and 90%) than a randomized baseline in identifying tests that lead to a fault and reduced the time spent running uninformative tests (speedup between 107% and 170%).

show abstract

Automated Identification and Qualitative Characterization of Safety Concerns Reported in UAV Software Platforms

Cited by 21 publications

References 128 publications

How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?

How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?

JUGE: An infrastructure for benchmarking Java unit test generators

Cost-effective Simulation-based Test Selection in Self-driving Cars Software with SDC-Scissor

Contact Info

Product

Resources

About