SWAT tool at the SBST 2021 Tool Competition

Humeniuk, Dmytro; Antoniol, Giuliano; Khomh, Foutse

doi:10.1109/sbst52555.2021.00019

Cited by 6 publications

(4 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Experiments: We design two sets of experiments as implemented in the SBST 2021 CPS testing tool competition. In order to provide a comparative analysis, we compare the results of the proposed test generators in Deeper with the presented test generators in the tool competition, that is, Frenetic, 24 GABExploit and GABExplore, 17 Swat, 25 and also the earlier version of Deeper 15 . We run the test generators on the test subject based on the same two experiment configurations as in the competition, which are shown in Table 1.…”

Section: Methodsmentioning

confidence: 99%

“…We provide a comparative analysis on the performance of the proposed bio‐inspired test generators in Deeper and five counterpart tools all integrated into the BeamNG simulator. In this regard, we compare the results of the proposed test generators in Deeper with five other test generator tools, Frenetic, 24 GABExploit and GABExplore, 17 Swat, 25 and also the earlier version of the Deeper (based on NSGA‐II) 15 —all participating tools in the CPS competition at SBST 2021. In order to do a fair comparison, we consider the same experimental evaluation procedures as the original CPS tool competition.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Machine learning testing in an ADAS case study using simulation‐integrated bio‐inspired search‐based testing

Moghadam

Borg

Saadatmand

et al. 2023

J Software Evolu Process

View full text Add to dashboard Cite

SummaryThis paper presents an extended version of Deeper, a search‐based simulation‐integrated test solution that generates failure‐revealing test scenarios for testing a deep neural network‐based lane‐keeping system. In the newly proposed version, we utilize a new set of bio‐inspired search algorithms, genetic algorithm (GA), and evolution strategies (ES), and particle swarm optimization (PSO), that leverage a quality population seed and domain‐specific crossover and mutation operations tailored for the presentation model used for modeling the test scenarios. In order to demonstrate the capabilities of the new test generators within Deeper, we carry out an empirical evaluation and comparison with regard to the results of five participating tools in the cyber‐physical systems testing competition at SBST 2021. Our evaluation shows the newly proposed test generators in Deeper not only represent a considerable improvement on the previous version but also prove to be effective and efficient in provoking a considerable number of diverse failure‐revealing test scenarios for testing an ML‐driven lane‐keeping system. They can trigger several failures while promoting test scenario diversity, under a limited test time budget, high target failure severity, and strict speed limit constraints.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Machine learning testing in an ADAS case study using simulation‐integrated bio‐inspired search‐based testing

Moghadam

Borg

Saadatmand

et al. 2023

J Software Evolu Process

View full text Add to dashboard Cite

show abstract

“…where Lev dist(i, j) indicates the weighted Levenshtein distance between the road segments. In order to provide a comparative analysis, we compare the results of the proposed test generators in Deeper with the presented test generators in the tool competition, i.e., Frenetic [25], GABExploit and GABExplore [22], Swat [26], and also the earlier version of Deeper [20]. We run the test generators on the test subject based on the same two experiment configurations as in the competition, which are shown in table I.…”

Section: Empirical Evaluationmentioning

confidence: 99%

“…In order to carry out an empirical evaluation, we use the setup provided by the cyber-physical systems (CPS) testing competition 1 We provide a comparative analysis on the performance of the proposed bio-inspired test generators in Deeper and five counterpart tools all integrated into the BeamNG simulator. In this regard, we compare the results of the proposed test generators in Deeper with five other test generator tools, Frenetic [25], GABExploit and GABExplore [22], Swat [26], and also the earlier version of the Deeper (based on NSGA-II) [20]-all participating tools in the CPS competition at SBST 2021. In order to do a fair comparison, we consider the same experimental evaluation procedures as the original CPS tool competition.…”

Section: Introductionmentioning

confidence: 99%

Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing

Moghadam¹,

Borg²,

Saadatmand³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper presents an extended version of Deeper, a search-based simulation-integrated test solution that generates failure-revealing test scenarios for testing a deep neural network-based lane-keeping system. In the newly proposed version, we utilize a new set of bio-inspired search algorithms, genetic algorithm (GA), (µ + λ) and (µ, λ) evolution strategies (ES), and particle swarm optimization (PSO), that leverage a quality population seed and domain-specific cross-over and mutation operations tailored for the presentation model used for modeling the test scenarios. In order to demonstrate the capabilities of the new test generators within Deeper, we carry out an empirical evaluation and comparison with regard to the results of five participating tools in the cyber-physical systems testing competition at SBST 2021. Our evaluation shows the newly proposed test generators in Deeper not only represent a considerable improvement on the previous version but also prove to be effective and efficient in provoking a considerable number of diverse failure-revealing test scenarios for testing an ML-driven lane-keeping system. They can trigger several failures while promoting test scenario diversity, under a limited test time budget, high target failure severity, and strict speed limit constraints.

show abstract

Frenetic-lib: An extensible framework for search-based generation of road structures for ADS testing

Klikovits

Castellano

Cetinkaya

et al. 2023

Science of Computer Programming

View full text Add to dashboard Cite

SWAT tool at the SBST 2021 Tool Competition

Cited by 6 publications

References 1 publication

Machine learning testing in an ADAS case study using simulation‐integrated bio‐inspired search‐based testing

Machine learning testing in an ADAS case study using simulation‐integrated bio‐inspired search‐based testing

Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing

Frenetic-lib: An extensible framework for search-based generation of road structures for ADS testing

Contact Info

Product

Resources

About