Search-based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions

Abstract

Search-based software testing (SBST) typically relies on fitness functions to guide the search exploration toward software failures. There are two main techniques to define fitness functions: (a) automated fitness function computation from the specification of the system requirements and (b) manual fitness function design. Both techniques have advantages. The former uses information from the system requirements to guide the search toward portions of the input domain that are more likely to contain failures. The latter uses the engineers' domain knowledge.We propose ATheNA, a novel SBST framework that combines fitness functions that are automatically generated from requirements specifications and manually defined by engineers. We design and implement ATheNA-S, an instance of ATheNA that targets Simulink ® models. We evaluate ATheNA-S by considering a large set of models and requirements from different domains. We compare our solution with an SBST baseline tool that supports automatically generated fitness functions, and another one that supports manually defined fitness functions. Our results show that ATheNA-S generates more failure-revealing test cases than the baseline tools and that the difference between the performance of ATheNA-S and the baseline tools is not statistically significant. We also assess whether ATheNA-S could generate failure-revealing test cases when applied to a large case study from the automotive domain. Our results show that ATheNA-S successfully revealed a requirement violation in our case study.

ARCH-COMP23 Category Report: Falsification

“…This competition enables maintaining this benchmark, which is getting traction in the falsification literature, maintained and updated. The participation of new tools (ATheNA [18], NNFal [28], and STGEM [34]) shows that this competition has been getting attention over the years. The tools now use completely different technologies that are challenging to compare.…”

“…Description. ATheNA [18] is a Matlab toolbox for automatic test case generation guided by a combination of automatic and manual fitness functions. ATheNA allows the user to specify a manually-defined fitness function and choose a strategy to combine the automatic and manual fitness values into the ATheNA fitness value.…”

“…This benchmark model provides a practical example from the medical domain, which the other models considered in the competition do not cover. • We had three new tools participating in the competition: ATheNA [18], NNFal [28], and STGEM [34] (see Section 3.6). The tools falsify [38] and FalStar [16] decided to not join the competition this year.…”