Practical Mutation Testing at Scale: A view from Google

Abstract

Mutation analysis assesses a test suite's adequacy by measuring its ability to detect small artificial faults, systematically seeded into the tested program. Mutation analysis is considered one of the strongest test-adequacy criteria. Mutation testing builds on top of mutation analysis and is a testing technique that uses mutants as test goals to create or improve a test suite. Mutation testing has long been considered intractable because the sheer number of mutants that can be created represents an insurmountable problem-both in terms of human and computational effort. This has hindered the adoption of mutation testing as an industry standard. For example, Google has a codebase of two billion lines of code and more than 150,000,000 tests are executed on a daily basis. The traditional approach to mutation testing does not scale to such an environment; even existing solutions to speed up mutation analysis are insufficient to make it computationally feasible at such a scale. To address these challenges, this paper presents a scalable approach to mutation testing based on the following main ideas:(1) mutation testing is done incrementally, mutating only changed code during code review, rather than the entire code base;(2) mutants are filtered, removing mutants that are likely to be irrelevant to developers, and limiting the number of mutants per line and per code review process; (3) mutants are selected based on the historical performance of mutation operators, further eliminating irrelevant mutants and improving mutant quality. This paper empirically validates the proposed approach by analyzing its effectiveness in a code-review-based setting, used by more than 24,000 developers on more than 1,000 projects. The results show that the proposed approach produces orders of magnitude fewer mutants and that context-based mutant filtering and selection improve mutant quality and actionability. Overall, the proposed approach represents a mutation testing framework that seamlessly integrates into the software development workflow and is applicable to industrial settings of any size.

Small-Amp: Test amplification in a dynamically typed language

“…Using Patterns to Guide Test Amplification As an important future work, we suggest using heuristics to guide the amplification algorithm. Large scale manual test amplification histories like the work at Google (Petrovic et al 2021;Petrović and Ivanković 2018) can be analyzed to answer questions like: How often do developers write new test methods? Are new test methods similar to an existing test?…”

“…Using mutation testing as an actionable target for strengthening an existing test suite is used at large scale at Google (Petrovic et al 2021;Petrović and Ivanković 2018) and Facebook (Beller et al 2021). These papers are close in spirit to DSPOT and SMALL-AMP as both create new test casess to increase mutation coverage.…”

Small-Amp: Test Amplification in a Dynamically Typed Language

“…Mutation Testing. Using mutation testing as an actionable target for strengthening an existing test suite is used at large scale at Google (Petrović et al 2021;Petrović and Ivanković 2018) and Facebook (Beller et al 2021). These papers are close in spirit to DSpot and Small-Amp as both create new test casess to increase mutation coverage.…”

“…As an important future work, we suggest using heuristics to guide the amplification algorithm. Large scale manual test amplification histories like the work at Google (Petrović et al 2021;Petrović and Ivanković 2018) can be analyzed to answer questions like: How often do developers write new test methods? Are new test methods similar to an existing test?…”

Mutation Testing of Quantum Programs: A Case Study With Qiskit

“…Mutation testing [19,20] has been shown to be an effective technique in improving testing practices, hence helping to guarantee program correctness. Big tech companies, such as Google and Facebook, have conducted several studies [21,22,23] advocating for mutation testing and its benefits. The general principle underlying mutation testing is that the bugs considered to generate buggy program versions represent realistic mistakes that programmers often make.…”