A Test-Suite Diagnosability Metric for Spectrum-Based Fault Localization Approaches

Abstract

Current metrics for assessing the adequacy of a testsuite plainly focus on the number of components (be it lines, branches, paths) covered by the suite, but do not explicitly check how the tests actually exercise these components and whether they provide enough information so that spectrum-based fault localization techniques can perform accurate fault isolation. We propose a metric, called DDU, aimed at complementing adequacy measurements by quantifying a test-suite's diagnosability, i.e., the effectiveness of applying spectrum-based fault localization to pinpoint faults in the code in the event of test failures. Our aim is to increase the value generated by creating thorough test-suites, so they are not only regarded as error detection mechanisms but also as effective diagnostic aids that help widely-used faultlocalization techniques to accurately pinpoint the location of bugs in the system. Our experiments show that optimizing a test suite with respect to DDU yields a 34% gain in spectrum-based fault localization report accuracy when compared to the standard branch-coverage metric.

Notice of Retraction: Does Refactoring of Test Smells Induce Fixing Flaky Tests?

“…The entire team relies on the results from these tests to decide on whether to merge a pull request [5] or to deploy the system [6], [7], [8], [9]. When it comes to testing, developer productivity is partly dependent on both (i) the ability of the tests to find real problems with the code being changed or developed [6], [10] and (ii) the cost of diagnosing the underlying cause in a timely and reliable fashion [11].…”

A Theoretical and Empirical Analysis of Program Spectra Diagnosability

“…We provide empirical evidence that optimizing a suite regarding DDU yields an increase of 34% in diagnostic accuracy when compared to test-suites that only consider branch-coverage as the optimization criterion and 17% when compared to optimizing mutation score. This paper extends our previous work [10] by (1) providing a generalization to the information-theoretic reasoning behind targeting a certain optimal spectrum density value, (2) providing a large-scale evaluation of DDU through a topology-based program spectra simulation -so that we are able to generate and analyze a vast breadth of qualitatively distinct faulty spectra -, (3) expanding our evaluation by comparing the diagnostic effectiveness of DDU versus mutation coverage, and (4) expanding our discussion on the implications of using the DDU metric for assessing diagnosability.…”

TBar: revisiting template-based automated program repair

“…Once the buggy position is localized, ARP tools can mutate the buggy code entity to generate patches. To identify defect locations in a program, several automated FL techniques have been proposed [72]: slice-based [47,71], spectrum-based [6,57], statistics-based [32,33], etc.…”