Killing Stubborn Mutants with Symbolic Execution

Chekam, Thierry Titcheu; Papadakis, Mike; Cordy, Maxime; Traon, Yves Le

doi:10.1145/3425497

Cited by 20 publications

(20 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…reliance on dedicated hardware, emulators, and simulators also prevents the use of static program analysis to detect equivalent mutants [21], [22], [23]. Such characteristics are common to embedded software in other types of CPS domains including avionics, automotive, and industry 4.0 (e.g., robotics systems).…”

Section: Textmentioning

confidence: 99%

Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results in the Space Domain

Olivares

Pastore

Briand

2022

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

On-board embedded software developed for spaceflight systems (space software) must adhere to stringent software quality assurance procedures. For example, verification and validation activities are typically performed and assessed by third party organizations. To further minimize the risk of human mistakes, space agencies, such as the European Space Agency (ESA), are looking for automated solutions for the assessment of software testing activities, which play a crucial role in this context. Though space software is our focus here, it should be noted that such software shares the above considerations, to a large extent, with embedded software in many other types of cyber-physical systems. Over the years, mutation analysis has shown to be a promising solution for the automated assessment of test suites; it consists of measuring the quality of a test suite in terms of the percentage of injected faults leading to a test failure. A number of optimization techniques, addressing scalability and accuracy problems, have been proposed to facilitate the industrial adoption of mutation analysis. However, to date, two major problems prevent space agencies from enforcing mutation analysis in space software development. First, there is uncertainty regarding the feasibility of applying mutation analysis optimization techniques in their context. Second, most of the existing techniques either can break the real-time requirements common in embedded software or cannot be applied when the software is tested in Software Validation Facilities, including CPU emulators and sensor simulators. In this paper, we enhance mutation analysis optimization techniques to enable their applicability to embedded software and propose a pipeline that successfully integrates them to address scalability and accuracy issues in this context, as described above. Further, we report on the largest study involving embedded software systems in the mutation analysis literature. Our research is part of a research project funded by ESA ESTEC involving private companies (GomSpace Luxembourg and LuxSpace) in the space sector. These industry partners provided the case studies reported in this paper; they include an on-board software system managing a microsatellite currently on-orbit, a set of libraries used in deployed cubesats, and a mathematical library certified by ESA.

show abstract

Section: Textmentioning

confidence: 99%

Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results in the Space Domain

Olivares

Pastore

Briand

2022

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…The benchmark contains a) mutants generated by Mart [37], a state-of-the-art tool that supports a comprehensive set of mutation operators and TCE7 [9], [38] on both pre-and post commit program versions of each commit, b) the mutant labels (whether they are commit-relevant), and c) large test pools created using a combination of test generation tools [35], [36], [39]. It is noted that the mutant test executions involved require excessive computational resources, i.e., require roughly 100 weeks of computation.…”

Section: A Benchmarks Usedmentioning

confidence: 99%

“…Another threat may relate to the mutants we use. To mitigate this threat, we selected data from a mutation testing tool [37] that has been used in several studies [16], [20], [39] that supports the most commonly used operators [50] and covers the most frequent features of the C language.…”

Section: B Feature Importancementioning

confidence: 99%

“…The test-based approximation of relevant and killable mu tants may introduce additional threats. To reduce it, we used test suites generated by KLEE [35] and SeMu [39], together with developer test suites.…”

Section: B Feature Importancementioning

confidence: 99%

“…Automatic test case generation aims at producing test inputs that a) make observable the code differences of two program versions [57], b) increase and optimize coverage [58] and kill mutants [39], [59], [60]. Among these techniques, the most relevant to our study are the are the ones related to patch testing, i.e., differential symbolic execution [61], KATCH [62] and Shadow symbolic execution [35].…”

Section: III Th R E a T S T O Va L I D I T Ymentioning

confidence: 99%

See 2 more Smart Citations

MuDelta: Delta-Oriented Mutation Testing at Commit Time

Chekam

Papadakis

et al. 2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

Self Cite

View full text Add to dashboard Cite

To effectively test program changes using mutation testing, one needs to use mutants that are relevant to the altered program behaviours. We introduce MuDelta, an approach that identifies commit-relevant mutants; mutants that affect and are affected by the changed program behaviours. Our approach uses machine learning applied on a combined scheme of graph and vector-based representations of static code features. Our results, from 50 commits in 21 Coreutils programs, demonstrate a strong prediction ability of our approach; yielding 0.80 (ROC) and 0.50 (PR-Curve) AUC values with 0.63 and 0.32 precision and recall values. These predictions are significantly higher than random guesses, 0.20 (PR-Curve) AUC, 0.21 and 0.21 precision and recall, and subsequently lead to strong relevant tests that kill 45% more relevant mutants than randomly sampled mutants (either sampled from those residing on the changed component(s) or from the changed lines). Our results also show that MuDelta selects mutants with 27% higher fault revealing ability in fault introducing commits. Taken together, our results corroborate the conclusion that commit-based mutation testing is suitable and promising for evolving software.

show abstract

On the use of commit-relevant mutants

Ojdanić

Laurent

et al. 2022

Empir Software Eng

View full text Add to dashboard Cite

Applying mutation testing to test subtle program changes, such as program patches or other small-scale code modifications, requires using mutants that capture the delta of the altered behaviours. To address this issue, we introduce the concept of commit-relevant mutants, which are the mutants that interact with the behaviours of the system affected by a particular commit. Therefore, commit-aware mutation testing, is a test assessment metric tailored to a specific commit. By analysing 83 commits from 25 projects involving 2,253,610 mutants in both C and Java, we identify the commit-relevant mutants and explore their relationship with other categories of mutants. Our results show that commit-relevant mutants represent a small subset of all mutants, which differs from the other classes of mutants (subsuming and hard-to-kill), and that the commit-relevant mutation score is weakly correlated with the traditional mutation score (Kendall/Pearson 0.15-0.4). Moreover, commit-aware mutation analysis provides insights about the testing of a commit, which can be more efficient than the classical mutation analysis; in our experiments, by analysing the same number of mutants, commit-aware mutants have better fault-revelation potential (30% higher chances of revealing commit-introducing faults) than traditional mutants. We also illustrate a possible application of commit-aware mutation testing as a metric to evaluate test case prioritisation.

show abstract

Killing Stubborn Mutants with Symbolic Execution

Cited by 20 publications

References 48 publications

Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results in the Space Domain

Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results in the Space Domain

MuDelta: Delta-Oriented Mutation Testing at Commit Time

On the use of commit-relevant mutants

Contact Info

Product

Resources

About