Empirical evaluation of the tarantula automatic fault-localization technique

Abstract

The high cost of locating faults in programs has motivated the development of techniques that assist in fault localization by automating part of the process of searching for faults. Empirical studies that compare these techniques have reported the relative effectiveness of four existing techniques on a set of subjects. These studies compare the rankings that the techniques compute for statements in the subject programs and the effectiveness of these rankings in locating the faults. However, it is unknown how these four techniques compare with Tarantula, another existing fault-localization technique, although this technique also provides a way to rank statements in terms of their suspiciousness. Thus, we performed a study to compare the Tarantula technique with the four techniques previously compared. This paper presents our study-it overviews the Tarantula technique along with the four other techniques studied, describes our experiment, and reports and discusses the results. Our studies show that, on the same set of subjects, the Tarantula technique consistently outperforms the other four techniques in terms of effectiveness in fault localization, and is comparable in efficiency to the least expensive of the other four techniques.

Effective Fault Localization Approach Using Feedback

“…To evaluate the effectiveness of our approach, the experimental study chooses the Siemens suite and Space as our benchmarks because they are two widely used benchmarks in the field of fault localization [2], [4], [8], [9], [11], [12], [15]- [21], [25]. They cover a wide spectrum of faults with high quality, such as predicate faults, assignment faults, missing code, etc.…”

“…SFL is independent of any specific model of system and incurs low time and space overhead. Due to the feature of statistics and its simplicity, SFL is widely accepted and studied as a promising technique in the fault localization community [2]- [5], [8]- [21], and the research [2], [3] has also empirically proven that SFL has high effectiveness of locating faults. Therefore, our approach is applied to SFL to fully demonstrate its effectiveness.…”

“…The performance of SFL is widely evaluated by the percentage of code that needs to be examined (or not examined) to find the fault [2]- [5], [8], [9], [11]- [15], [17]- [21]. This evaluation assumes that debugging engineers will examine all statements from top to bottom according to the ranking list given by the SFL until they encounter the faulty statement.…”

“…However, debugging is one of the most time-consuming tasks in the development and maintenance of software [1]. A typical process of debugging consists of four steps: failure reproduction, fault localization, fault repair and repair verification, among which fault localization is usually a tedious and difficult step [2]. With the aim at decreasing the cost of debugging, many researchers try to optimize the process of fault localization and improve its performance [1]- [25].…”

“…Typically, SFL exploits the correlations between program entities and program failures by statistically analyzing coverage information. SFL has been widely studied and used in the field of fault localization [2]- [5], [8]- [21], and the prior research [2], [3] has empirically proven that SFL yields a promising ability in locating faults.…”