Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results in the Space Domain

Abstract

On-board embedded software developed for spaceflight systems (space software) must adhere to stringent software quality assurance procedures. For example, verification and validation activities are typically performed and assessed by third party organizations. To further minimize the risk of human mistakes, space agencies, such as the European Space Agency (ESA), are looking for automated solutions for the assessment of software testing activities, which play a crucial role in this context. Though space software is our focus here, it should be noted that such software shares the above considerations, to a large extent, with embedded software in many other types of cyber-physical systems. Over the years, mutation analysis has shown to be a promising solution for the automated assessment of test suites; it consists of measuring the quality of a test suite in terms of the percentage of injected faults leading to a test failure. A number of optimization techniques, addressing scalability and accuracy problems, have been proposed to facilitate the industrial adoption of mutation analysis. However, to date, two major problems prevent space agencies from enforcing mutation analysis in space software development. First, there is uncertainty regarding the feasibility of applying mutation analysis optimization techniques in their context. Second, most of the existing techniques either can break the real-time requirements common in embedded software or cannot be applied when the software is tested in Software Validation Facilities, including CPU emulators and sensor simulators. In this paper, we enhance mutation analysis optimization techniques to enable their applicability to embedded software and propose a pipeline that successfully integrates them to address scalability and accuracy issues in this context, as described above. Further, we report on the largest study involving embedded software systems in the mutation analysis literature. Our research is part of a research project funded by ESA ESTEC involving private companies (GomSpace Luxembourg and LuxSpace) in the space sector. These industry partners provided the case studies reported in this paper; they include an on-board software system managing a microsatellite currently on-orbit, a set of libraries used in deployed cubesats, and a mathematical library certified by ESA.

Data-Driven Mutation Analysis for Cyber-Physical Systems

“…Indeed, in the worst case (i.e., ESAIL-ADCS), mutation analysis can be performed in 9 hours with 100 parallel computation nodes; in safety-critical contexts, where development entails large costs, buying computation time on the Cloud is affordable. Code-driven mutation analysis for systems with similar characteristics lasts considerably more [56], [57]. For ESAIL and LIBP, for example, mutants sampling 6 makes code-driven mutation analysis feasible with parallel computation nodes, leading to 1800 hours (ESAIL) and 3 hours (LIBP) of execution time, which is still significantly higher than the time required by DaMAT.…”

“…Mutants sampling consists of selecting a subset of mutants to compute the mutation score [58]. In our previous work [57], we have demonstrated that a fixed-width sequential confidence interval approach guarantees the accurate estimation of the mutation score. ever, although useful to derive an accurate mutation score, mutants sampling prevents the complete identification of test suite shortcomings.…”

“…For example, to generate unit test cases with KLEE [61], which is a well-known test generation tool based on symbolic execution, it is necessary to implement test templates distinguishing the outputs and the inputs to be treated symbolically for each function under test, which may entail substantial effort for large software systems. Also, to perform code-driven mutation analysis with MASS [57], which is our recent code-driven mutation analysis tool for CPS software, it is necessary to update the configuration and execution scripts for the SUT (e.g., to collect code coverage data, compile with multiple compiler optimization options, determine test timeouts, and track test failures), an activity that requires a few hours to complete. Further, other mutation analysis tools require manual effort to be executed; for example, MART [62] and Mull [63] require the software to be compiled with LLVM [64], which implies modifying compilation scripts and often leads to compilation errors that are hard to solve for large CPS software with many dependencies on third party libraries [57].…”

“…Also, to perform code-driven mutation analysis with MASS [57], which is our recent code-driven mutation analysis tool for CPS software, it is necessary to update the configuration and execution scripts for the SUT (e.g., to collect code coverage data, compile with multiple compiler optimization options, determine test timeouts, and track test failures), an activity that requires a few hours to complete. Further, other mutation analysis tools require manual effort to be executed; for example, MART [62] and Mull [63] require the software to be compiled with LLVM [64], which implies modifying compilation scripts and often leads to compilation errors that are hard to solve for large CPS software with many dependencies on third party libraries [57]. Finally, note that our implementation of DaMAT enables engineers to introduce probes into the SUT source code using dedicated annotation labels (i.e., specific comment keywords); such annotations do not alter the execution of production code but can be used to automatically perform mutation analysis within a CI/CD pipeline without requiring manual intervention.…”

“…Finally, based on our experience, data-driven mutation analysis leads to a lower number of mutants than traditional code-driven mutation analysis. For example, in the case of LIBP, DaMAT generates 44 mutants while codedriven mutation analysis, based on our previous work [57], leads to 3931 mutants (346, if mutants sampling is applied). The low number of mutants generated by DaMAT leads to a low number of mutants to be inspected (e.g., because not detected by the test suite), which greatly reduces the cost of the analysis.…”