A Critical Evaluation of Spectrum-Based Fault Localization Techniques on a Large-Scale Software System

Abstract

Abstract-In the past, spectrum-based fault localization (SBFL) techniques have been developed to pinpoint a fault location in a program given a set of failing and successful test executions. Most of the algorithms use similarity coefficients and have only been evaluated on established but small benchmark programs from the Software-artifact Infrastructure Repository. In this paper, we evaluate the feasibility of applying 33 state-ofthe-art SBFL techniques to a large real-world project, namely ASPECTJ. From an initial set of 350 faulty version from the iBugs repository of ASPECTJ we manually classified 88 bugs where SBFL techniques are suitable. Notably, only 11 bugs of these bugs can be found after examining the 1000 most suspicious lines and on average 250 source code files need to be inspected per bug. Based on these results, the study showcases the limitations of SBFL on a larger program and points out areas for improvement.

An evaluation of pure spectrum‐based fault localization techniques for large‐scale software systems

“…In turn, this jeopardizes the practical effectiveness of SBFL techniques . The results of our study show that only 11 bugs can be detected in the best case by any of the investigated SBFL approaches after examining the top 1000 suspicious lines in the ranking. Furthermore, it would require on average 250 files that need to be inspected to discover any bug.…”

“…In the sequel, we briefly introduce more recent ranking metrics that are not included in the set of traditional ranking metrics, serving as baseline for the evaluation presented by Naish et al and which were not considered in our previous work …”

“…In our previous work, we claim that an evaluation based only on small projects is insufficient to derive effectiveness results of SBFL techniques. Thus, we evaluated the existing pure SBFL approaches on a large‐scale software application namely A spect J via 350 bugs from the iBugs repository …”

“…To confirm our previous results and to complement the existing theoretical and experimental evaluations of SBFL metrics, this paper provides the following contributions: We provide an evaluation of pure SBFL approaches based on two large‐scale benchmarks, namely A spect J via the iBugs repository and D efects4 J, where we specifically cross compare the results between the two benchmarks. We evaluate and compare SBFL approaches based on absolute metrics, such as the wasted effort (WE), the number of bugs found after inspecting X ranked element ( Hit@X ), the number of files investigated (NFI), as well as area between curves (ABC) metrics, which provide insights into the uncertainty of the provided ranking. In our evaluations, we additionally include the current‐state‐of‐the‐art SBFL metrics, namely Naish2 (Op2), D ∗ , the GP‐evolved metric GP13, and hyperbolic functions . These metrics have not been covered by our previous evaluation …”

“…Section 2 overviews the background principles and definitions of SBFL and defines the metrics that will be evaluated and compared, as well as the criteria they have been originally evaluated on. Section 3 describes the design of our study, also summarizing our previous preliminary results . Section 4 describes the A spect J and D efects4 J case study, including the data collection process and the classification of the faults present in the code base.…”