A practical guide to select quality indicators for assessing pareto-based search algorithms in search-based software engineering

Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering

et al. 2016

169

163

Recent years have seen a proliferation of complex Advanced Driver Assistance Systems (ADAS), in particular, for use in autonomous cars. These systems consist of sensors and cameras as well as image processing and decision support software components. They are meant to help drivers by providing proper warnings or by preventing dangerous situations. In this paper, we focus on the problem of design time testing of ADAS in a simulated environment. We provide a testing approach for ADAS by combining multiobjective search with surrogate models developed based on neural networks. We use multi-objective search to guide testing towards the most critical behaviors of ADAS. Surrogate modeling enables our testing approach to explore a larger part of the input search space within limited computational resources. We characterize the condition under which the multi-objective search algorithm behaves the same with and without surrogate modeling, thus showing the accuracy of our approach. We evaluate our approach by applying it to an industrial ADAS system. Our experiment shows that our approach automatically identifies test cases indicating critical ADAS behaviors. Further, we show that combining our search algorithm with surrogate modeling improves the quality of the generated test cases, especially under tight and realistic computational resources.

Section: Discussionmentioning

confidence: 99%

“…The HV and GD are selected from the combination and convergence quality indicator categories, respectively. As discussed in [53], to assess the quality of computed pareto fronts with respect to combination and convergence indicators, it is su cient to choose only one indicator from each of these two categories. Experiment Design.…”

Section: Discussionmentioning

confidence: 99%

Testing advanced driver assistance systems using multi-objective search and neural networks

Abdessalem

Nejati

Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering

et al. 2016

169

163

“…To compute the quality indicators, following existing guidelines in the literature [33], we compute a reference Pareto front as the union of all the nondominated solutions obtained from all runs of NSGAII-DT and NSGAII. The HV quality indicator [38] measures the size of the space covered by the members of a Pareto front generated by a search algorithm.…”

Section: Metricsmentioning

confidence: 99%

Testing vision-based control systems using learnable evolutionary algorithms

Abdessalem

Nejati

Proceedings of the 40th International Conference on Software Engineering

et al. 2018

214

173

Vision-based control systems are key enablers of many autonomous vehicular systems, including self-driving cars. Testing such systems is complicated by complex and multidimensional input spaces. We propose an automated testing algorithm that builds on learnable evolutionary algorithms. These algorithms rely on machine learning or a combination of machine learning and Darwinian genetic operators to guide the generation of new solutions (test scenarios in our context). Our approach combines multiobjective populationbased search algorithms and decision tree classification models to achieve the following goals: First, classification models guide the search-based generation of tests faster towards critical test scenarios (i.e., test scenarios leading to failures). Second, search algorithms refine classification models so that the models can accurately characterize critical regions (i.e., the regions of a test input space that are likely to contain most critical test scenarios). Our evaluation performed on an industrial automotive vision-based control system shows that: (1) Our algorithm outperforms a baseline evolutionary search algorithm and generates 78% more distinct, critical test scenarios compared to the baseline algorithm. (2) Our algorithm accurately characterizes critical regions of the system under test, thus identifying the conditions that likely to lead to system failures.

“…We selected NSGA-II because it has been widely used in the software engineering community to address problems with multiple objectives [55], e.g., regression testing [30]. In addition, it is Table III AN EXAMPLE OF TWO CANDIDATE SOLUTIONS ENCODED AS CHROMOSOMES BASED ON THE PATH CONDITIONS AND SLICES DEPICTED IN TABLE II.…”

Section: Tailoring Nsga-ii To the Waf Fixing Problemmentioning

confidence: 99%

Automatically Repairing Web Application Firewalls Based on Successful SQL Injection Attacks

Appelt

Panichella

2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE)

2017

Abstract-Testing and fixing Web Application Firewalls (WAFs) are two relevant and complementary challenges for security analysts. Automated testing helps to cost-effectively detect vulnerabilities in a WAF by generating effective test cases, i.e., attacks. Once vulnerabilities have been identified, the WAF needs to be fixed by augmenting its rule set to filter attacks without blocking legitimate requests. However, existing research suggests that rule sets are very difficult to understand and too complex to be manually fixed. In this paper, we formalise the problem of fixing vulnerable WAFs as a combinatorial optimisation problem. To solve it, we propose an automated approach that combines machine learning with multi-objective genetic algorithms. Given a set of legitimate requests and bypassing SQL injection attacks, our approach automatically infers regular expressions that, when added to the WAF's rule set, prevent many attacks while letting legitimate requests go through. Our empirical evaluation based on both open-source and proprietary WAFs shows that the generated filter rules are effective at blocking previously identified and successful SQL injection attacks (recall between 54.6% and 98.3%), while triggering in most cases no or few false positives (false positive rate between 0% and 2%).