Evaluation of Mutant Sampling Criteria in Object-Oriented Mutation Testing

Abstract

Abstract-Mutation testing of object-oriented programs differs from that of standard (traditional) mutation operators in accordance to the number of generated mutants and ability of tests to kill mutants. Therefore, outcomes of cost reduction analysis cannot be directly transferred from a standard mutation to an object-oriented one. Mutant sampling is one of reduction methods of the number of generated and tested mutants. We proposed different mutant sampling criteria based on equivalence partitioning in respect to object-oriented program features. The criteria were experimentally evaluated for object-oriented and standard mutation operators applied in C# programs. We compared results using a quality metric, which combines mutation score accuracy with mutation cost factors. In result, class random sampling and operator random sampling are recommended for OO and standard mutation testing, accordingly. With a reasonable decline of result accuracy, the mutant sampling technique is easily applicable in comparison to other cost reduction techniques.

Cost measures matter for mutation testing study validity

“…Six papers acknowledge that this is a threat [20,24,29,34,59, 93] and five of those try to deal with it [20,24,29,34,59], but none of them evaluated the degree to which this threat affects scientific conclusions of previous work. Only three [20,24,34] of the 15 papers that measure the time also acknowledge the threat of using number of mutants as a surrogate for execution time. The other 12 papers implicitly mitigate such threat by simply measuring and reporting the execution time.…”

“…RMS is arguably the most naive type of mutant reduction approach, as mutants are discarded with no particular preference, thus representing an unbiased reduction. This strategy has also served as a baseline comparison for several other work in the literature [24,53,94,97]. Therefore, assessing the error for this type of strategy provides a meaningful overview of what to expect from mutant reduction strategies in general.…”

“…Year Venue Mutation Tool Metrics Strategies Ack Mitigates #M Time RMS SM Other Al-Hajjaji et al [1] 2017 VACE Proteum X X X Ammann et al [2] 2014 ICST Proteum X X X Barbosa et al [4] 2001 JSTVR Proteum X X Bluemke and Kulesza [5] 2013 AISC MuJava X X Bluemke and Kulesza [7] 2014 AISC MuJava X X Bluemke and Kulesza [6] 2014 ICSOFT-EA MuJava X X Delamaro et al [12] 2014 ICST Proteum X X Delamaro et al [14] 2014 ICST Proteum X X Delamaro et al [13] 2014 SBES Proteum X X X Delgado-Pérez et al [16] 2017 JSTVR MuCpp X X Delgado-Pérez et al [15] 2017 IST MuCpp X X X Deng et al [18] 2013 ICST Mothra X X Derezińska and Rudnik [23] 2012 TOOLS CREAM X X X X Derezińska [19] 2013 AISC CREAM X X X Derezińska and Hałas [21] 2014 ICSTW MuPy X X X X Derezińska and Hałas [22] 2015 AISC MuPy X X X X Derezińska [20] 2016 AISC CREAM X X X X X Measures execution time Derezińska and Rudnik [24] 2017 FedCSIS CREAM X X X X Measures execution time Domínguez-Jiménez et al [25] 2011 IST GAmera X X X Gligoric et al [29] 2013 ISSTA Comutation X X X X Measures the number of states Gong et al [30] 2017 IST MuJava X X X Gopinath et al [32] 2015 ISSRE PIT X X X Gopinath et al [33] 2016 ICSE PIT X X X Gopinath et al [31] 2017 TR PIT X X X X Guizzo et al [34] 2020 TSE PIT X X X X X X Measures execution time Harman et al [36] 2014 ASE Bacterio X X Iida and Takada [38] 2017 Mutation MuJava X X X Inozemtseva et al [39] 2013 FSE GiGAn X X X Ji et al [40] 2009 SEKE MuJava X X Just and Schweiggert [44] 2014 JSTVR Major X X X Just et al [43] 2017 ISSTA Major X X Kaminski and Ammann [45] 2009 ICST TRF-TIF X X Kaminski et al [47] 2011 IST TRF-TIF X X X Kaminski et al…”

“…As we showed in Section 2, 60 out of 75 evaluated papers use either one or the other. Furthermore, they both are commonly used as a baseline for comparison in mutant reduction work [24,53,94,97].…”

“…Despite being a known threat to validity [20,24,29,59,93], researchers usually report the cost reduction in terms of % fewer mutants and do not measure the reduction in execution time. As described in Section 2, out of 75 analysed papers, only 15 (20%) report execution time, and 54 of them (72%) are vulnerable to the threat presented in this work.…”