On the Accuracy of Spectrum-based Fault Localization

Abstract

Spectrum-based fault localization shortens the testdiagnose-repair cycle by reducing the debugging effort. As a lightweight automated diagnosis technique it can easily be integrated with existing testing schemes. However, as no model of the system is taken into account, its diagnostic accuracy is inherently limited. Using the Siemens Set benchmark, we investigate this diagnostic accuracy as a function of several parameters (such as quality and quantity of the program spectra collected during the execution of the system), some of which directly relate to test design. Our results indicate that the superior performance of a particular similarity coefficient, used to analyze the program spectra, is largely independent of test design. Furthermore, nearoptimal diagnostic accuracy (exonerating about 80% of the blocks of code on average) is already obtained for low-quality error observations and limited numbers of test cases. The influence of the number of test cases is of primary importance for continuous (embedded) processing applications, where only limited observation horizons can be maintained.

Prioritizing tests for software fault diagnosis

“…In the previous literature, interface, type, and variable declarations are considered in the component ranking and the C d metric, although their likelihoods are in most cases 0 because of the limitations on the code instrumentation, which causes ∀ ia ij = 0. This is especially true in Spectrum‐based techniques 9–11. As they are located at the bottom of the ranking, the number of inspected components (the numerator in the diagnostic effort formula used in Section 7.1) does not change.…”

“…Although there is a large number of different diagnosis techniques (see Section 9), our work is based on Bayesian diagnosis, well known from Model‐Based Diagnosis, an area within AI. Compared to other, statistical approaches, such as Tarantula 11, Ochiai 9, and alternative techniques 19–25, Bayesian diagnosis is founded on probability theory, and is the only technique that can serve as the base for our test prioritization heuristic search function, described in Section 4.…”

“…Automated fault‐localization techniques also aim at minimizing diagnostic cost when failures occur during the testing phase. Statistical approaches include the Tarantula tool by Jones et al 11, Ochiai by Abreu et al 9, the Nearest Neighbor technique by Renieris et al 22, Sober by Liu et al 21, CBI by Liblit and his colleagues 20, and CrossTab by Wang et al 25. Approaches to statistical fault localization need not be limited to the statement level, work that considers execution paths and dependencies includes 19, 23, 24, 43.…”

“…The debugging phase can make use of automatic fault localization techniques, which help to significantly reduce the manual debugging effort needed, as shown in 9–12. Fault localization algorithms use the information provided by tests executed in the testing phase to deduce a list of program elements (e.g.…”

An evaluation of pure spectrum‐based fault localization techniques for large‐scale software systems

“…Xie et al provided a theoretical evaluation of the original SBFL approaches investigated by Naish et al and approaches that have been derived by a genetic algorithm . However, most of these metrics have originally been evaluated and have subsequently been compared on relatively small‐scale artifacts from the publicly available Software‐artifact Infrastructure Repository (SIR) . Only recently, some SBFL techniques have been evaluated on programs from the D efects4 J benchmark …”