CRISCE: Towards Generating Test Cases from Accident Sketches

Nguyen, Vuong; Gambi, Alessio; Ahmed, Jasim

doi:10.1109/icse-companion55297.2022.9793783

Cited by 1 publication

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The choice of the test subject system is a potential threat to the external validity of the results. However, the test system in this study is one of the commonly used systems in self‐driving cars, 47–49 and furthermore, different ML models with various quality levels (i.e., different accuracy levels) could be deployed within the BeamNG.AI agent, 22 and in this regard, the proposed test scenario generation techniques can still be used. Nonetheless, it still offers one type of DL‐based systems in self‐driving cars, and further studies are required to address the testing of other DL‐driven systems; meanwhile, we also keep the tool open for extensions, for example, to support the execution of test scenarios in other simulators.…”

Section: Resultsmentioning

confidence: 99%

Section: External Validitymentioning

confidence: 99%

“…Furthermore, with regard to the use of BeamNG.AI as the test subject and BeamNG.Tech as the simulation environment in ML testing, in particular, ADAS testing, Nguyen et al 47 propose a test generator that uses accident sketches and with the help of image processing identifies the road segments, lane markings, and vehicle positions and generates the corresponding test scenarios in BeamNG simulator. Birchler et al 48 develop a test scenario selection (SDC‐Scissor) that uses ML techniques, that is, logistic regression, random forest, and naive Bayes, to train ML models to identify safe and unsafe test scenarios before their execution.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Machine learning testing in an ADAS case study using simulation‐integrated bio‐inspired search‐based testing

Moghadam

Borg

Saadatmand

et al. 2023

J Software Evolu Process

View full text Add to dashboard Cite

SummaryThis paper presents an extended version of Deeper, a search‐based simulation‐integrated test solution that generates failure‐revealing test scenarios for testing a deep neural network‐based lane‐keeping system. In the newly proposed version, we utilize a new set of bio‐inspired search algorithms, genetic algorithm (GA), and evolution strategies (ES), and particle swarm optimization (PSO), that leverage a quality population seed and domain‐specific crossover and mutation operations tailored for the presentation model used for modeling the test scenarios. In order to demonstrate the capabilities of the new test generators within Deeper, we carry out an empirical evaluation and comparison with regard to the results of five participating tools in the cyber‐physical systems testing competition at SBST 2021. Our evaluation shows the newly proposed test generators in Deeper not only represent a considerable improvement on the previous version but also prove to be effective and efficient in provoking a considerable number of diverse failure‐revealing test scenarios for testing an ML‐driven lane‐keeping system. They can trigger several failures while promoting test scenario diversity, under a limited test time budget, high target failure severity, and strict speed limit constraints.

show abstract

Section: Resultsmentioning

confidence: 99%