Efficient multi-objective higher order mutation testing with genetic programming

Abstract

It is said ninety percent of faults that survive manufacturer's testing procedures are complex. That is, the corresponding bug fix contains multiple changes. Higher order mutation testing is used to study defect interactions and their impact on software testing for fault finding. We adopt a multi-objective Pareto optimal approach using Monte Carlo sampling, genetic algorithms and genetic programming to search for higher order mutants which are both hard-to-kill and realistic. The space of complex faults (higher order mutants) is much larger than that of traditional first order mutations which correspond to simple faults, nevertheless search based approaches make this scalable. The problems of non-determinism and efficiency are overcome. Easy to detect faults may become harder to detect when they interact and impossible to detect single faults may be brought to light when code contains two such faults. We use strong typing and BNF grammars in search based mutation testing to find examples of both in ancient heavily optimised every day C code.

Creating and Debugging Performance CUDA C

“…Remember to include code to check the kernel really is working. Once satisfied with your first kernel, inject a fault into it [22]. Did it fail in the way you expected?…”

Mutant reduction based on dominance relation for weak mutation testing

“…Jia and Harman utilized meta-heuristic search algorithms to generate semantic HOMs which are hard to kill, and reduced mutants greatly [4]. Langdon et al combined FOMs with close semantic relations to form HOMs [31]. Second-order mutants (SOMs, two syntactic changes in a program) proposed by Polo et al get 50% cost saving [32].…”

“…The program, Triangle, is a familiar example in software testing [2,4,30,31], and Figure 1 (a) is its Java version. Figure 1 shows the process of forming P where (a) is the code of P (Triangle); (b) represents the CFG of code from lines 7 to 9 in the dashed box of (a); (c) means the mutant branches constructed after performing ROR on "i f (a == b)" and AORB on "trian = trian + 1", respectively; and (d) is the CFG after inserting mutant branches into P. The dashed boxes in Figure 1 (d) represent the inserted mutant branches.…”

“…Among these programs, J1 is from [2], [4], [30] and [31]; J2, J3 and J4 are classical mutation testing programs appeared in [3]; J5 and J6 are example programs in [50], and J5, J6 and J7 are experimental programs in [17]; J8, J9 and J10 are from [51]. All these are programmed in Java.…”

Achievements, Open Problems and Challenges for Search Based Software Testing

“…However, for test data generation problems, the large majority of existing approaches are single objective. Relatively few attack multi-objective test case generation [7], [34], [74], [84], [124], despite it having been proposed sometime ago [52]. This is unrealistic because practising software testers are unlikely to be concerned only with a single test objective [45].…”