HOTFUZ: Cost‐effective higher‐order mutation‐based fault localization

Abstract

Fault localization techniques are used to deduce the exact source of a failure from a set of failure indications while debugging software and play a crucial role in improving software quality. Mutation‐based fault localization (MBFL) techniques are proposed to localize faults at a finer granularity and with higher accuracy than traditional fault localization techniques. Despite the technique's effectiveness, the immense cost of mutation analysis hinders MBFL's practical application in the industry. Various mutation alternative strategies are utilized to lower the cost of MBFL, but they sacrifice the accuracy of localization results. Higher‐order mutation testing was proposed to search for valuable mutants that drive testing harder and reduce the overall test effort. However, higher‐order mutants (HOMs) never have been used to address the cost problem of MBFL to the extent of our knowledge. This paper proposes a novel, cost‐effective MBFL technique called HOTFUZ, Higher‐Order muTation‐based FaUlt localiZation, that employs HOMs to reduce the cost while minimizing the accuracy degradation. HOTFUZ combines mutants of a program under test into HOMs to decrease the number of mutants by more than half, depending on the order of HOMs. An experimental study is conducted using 65 real‐world faults of CoREBench to assess the proposed approach's cost‐effectiveness. The experimental results show that HOTFUZ outperforms the extant mutation alternative strategies by localizing faults more accurately using the same number of mutants executed. HOTFUZ has three main benefits over existing mutant reduction techniques for MBFL: (a) It keeps the advantage of using the whole set of mutation operators; (b) it does not discard generated mutants randomly for the sake of efficiency; and, finally, (c) it significantly decreases the proportion of equivalent mutants.

Delta4Ms: Improving mutation‐based fault localization by eliminating mutant bias

“…HOM testing regards HOMs as representations of complex faults which can alleviate the of complex mutation testing challenge. For instance, HOMs are valuable in reducing testing costs [34][35][36][37], assessing the quality of test suites [38], facilitating test data generation [39], performing coupling effect analysis [19,40] and executing fault localization [41,42].…”

Learning test-mutant relationship for accurate fault localisation

“…MBFL techniques suffer from the cost of mutation analysis, as it requires a massive number of test executions against the generated mutants. To reduce the number of mutants needed to be inspected, Metallaxis [13] adopts mutant sampling while HOTFUZ [18] combines first-order mutants to build higher-order mutants and computes suspicious score only on them. Their results showed that this reduces the number of total mutants as well as equivalent mutants that contribute nothing to the fault localisation.…”

“…Although the usefulness of MBFL techniques has long been demonstrated, the substantial cost of mutation analysis has hampered their practical applications. The cost reduction techniques for MBFL are in line with the cost reduction techniques for traditional mutation testing: random sampling of the mutants [13] and higher-order mutants [18]. They have reduced the number of mutants and test executions, but still they are not suitable for large systems like Google's on which more than ten million test executions take place everyday [19].…”

GaSubtle: A New Genetic Algorithm for Generating Subtle Higher-Order Mutants

“…Jang et al [41] proposed a higher-order mutation-based fault localization technique called HOTFUZ. The approach uses HOMs to reduce the cost while minimizing the accuracy degradation.…”