Assurances in Software Testing: A Roadmap

Abstract

As researchers, we already understand how to make testing more effective and efficient at finding bugs. However, as fuzzing (i.e., automated testing) becomes more widely adopted in practice, practitioners are asking: Which assurances does a fuzzing campaign provide that exposes no bugs? When is it safe to stop the fuzzer with a reasonable residual risk? How much longer should the fuzzer be run to achieve sufficient coverage?It is time for us to move beyond the innovation of increasingly sophisticated testing techniques, to build a body of knowledge around the explication and quantification of the testing process, and to develop sound methodologies to estimate and extrapolate these quantities with measurable accuracy. In our vision of the future practitioners leverage a rich statistical toolset to assess residual risk, to obtain statistical guarantees, and to analyze the cost-benefit trade-off for ongoing fuzzing campaigns. We propose a general framework as a first starting point to tackle this fundamental challenge and discuss a large number of concrete opportunities for future research. 4 In the quotes, the emphasis in italic letters is mine.

Mutation Analysis: Answering the Fuzzing Challenge

“…One of the promising approaches toward estimation of equivalent mutants is using species richness estimation as proposed by Böhme. [91]. The idea is to use the frequency of counts of mutants that are found by different test cases to estimate the number of mutants that are yet to be found.…”

Testing self-adaptive software with probabilistic guarantees on performance metrics

“…In the web-application case mentioned above, this parameter could be, for example, the average time spent recovering from server failures over the whole test duration. The key intuition is that this performance parameter is itself a random variable [9], and we can therefore use tools from statistics to deduce properties of its value.…”

“…The approach we propose in this paper is based on random sampling of the system inputs and environment scenarios. This practice is known to the software testing community [3,16], and is often called fuzz testing [9,64,71,73]. The literature focuses on using random generation for achieving adequate exploration of the software behaviour, e.g., code coverage [64].…”

“…Contribution: In this paper we address the three mentioned research challenges by leveraging a rigorous probabilistic approach [9,22]. The probabilistic approach is beneficial in two ways: (i) it allows the efficient exploration of large input and configuration spaces [54], and (ii) it can provide a quantification of its own adequacy.…”

“…As a consequence, the measured performance metric must be treated as a random quantity, and requires statistical evaluation. We therefore enter the domain of probabilistic guarantees [1,9].…”