Angels and monsters

Abstract

We study the simultaneous test effectiveness and efficiency improvement achievable by Strongly Subsuming Higher Order Mutants (SSHOMs), constructed from 15,792 first order mutants in four Java programs. Using SSHOMs in place of the first order mutants they subsume yielded a 35%-45% reduction in the number of mutants required, while simultaneously improving test efficiency by 15% and effectiveness by between 5.6% and 12%. Trivial first order faults often combine to form exceptionally non-trivial higher order faults; apparently innocuous angels can combine to breed monsters. Nevertheless, these same monsters can be recruited to improve automated test effectiveness and efficiency.

Efficiently finding higher-order mutants

“…In addition to using search var to get a complete set of SSHOMs with regard to given tests and first-order mutants, we compare the efficiency and effectiveness of search var against the existing stateof-the-art genetic search (search gen ) and a baseline brute-force strategy (search bf ), based on subject systems previously used in evaluating the genetic search strategy [17]. Subject Systems.…”

“…Subject Systems. We replicate the setup of the largest previous study on higher-order mutation testing [17]. While we cannot perform an exact replication since we could not obtain the original tools from the authors, not all relevant details and parameters have been published, and some engineering limitations discussed earlier, we still select the same subject systems and reimplement search strategies in our own infrastructure.…”

“…A recent survey of over 39 papers on higher-order mutation testing [13] summarized a large number of different application scenarios for higher-order mutants claimed in prior research, including mutant reduction [12,17,19], coupling effect analysis [14,23], equivalent mutant reduction [37,49], test data evaluation [16], and test suite reduction [17,52]. In the following, we illustrate a concrete example of how higher-order mutations can be useful to softwareengineering researchers for creating synthetic, but challenging faults to evaluate various software engineering tools.…”

“…However, SSHOMs are tricky to find among the vast quantity of possible combinations of first-order mutants. Current approaches use genetic-search techniques, guided by a simple fitness function [17,21,23,43]. Since SSHOMs are difficult to find, little is known about them and their characteristics.…”

Search-based mutant selection for efficient test suite improvement: Evaluation and results

“…More specifically, these techniques have been used to find subsuming HOMs [35,36], that is, HOMs that are more difficult to kill than their constituent FOMs. Harman et al [37] provided a more restrictive fitness function to find strongly subsuming high order mutants (SSHOMs). Lima et al [38] recently compared HOM-based strategies and traditional strategies, such as random mutant selection, selective mutation and search-based mutant selection (genetic algorithm).…”