Will Fault Localization Work for These Failures? An Automated Approach to Predict Effectiveness of Fault Localization Tools

Abstract

Abstract-Debugging is a crucial yet expensive activity to improve the reliability of software systems. To reduce debugging cost, various fault localization tools have been proposed. A spectrum-based fault localization tool often outputs an ordered list of program elements sorted based on their likelihood to be the root cause of a set of failures (i.e., their suspiciousness scores). Despite the many studies on fault localization, unfortunately, however, for many bugs, the root causes are often low in the ordered list. This potentially causes developers to distrust fault localization tools. Recently, Parnin and Orso highlight in their user study that many debuggers do not find fault localization useful if they do not find the root cause early in the list.To alleviate the above issue, we build an oracle that could predict whether the output of a fault localization tool can be trusted or not. If the output is not likely to be trusted, developers do not need to spend time going through the list of most suspicious program elements one by one. Rather, other conventional means of debugging could be performed. To construct the oracle, we extract the values of a number of features that are potentially related to the effectiveness of fault localization. Building upon advances in machine learning, we process these feature values to learn a discriminative model that is able to predict the effectiveness of a fault localization tool output. In this preliminary work, we consider an output of a fault localization tool to be effective if the root cause appears in the top 10 most suspicious program elements. We have experimented our proposed oracle on 200 faulty programs from Space, NanoXML, XML-Security, and the 7 programs in Siemens test suite. Our experiments demonstrate that we could predict the effectiveness of fault localization tool with a precision, recall, and F-measure (harmonic mean of precision and recall) of 54.36%, 95.29%, and 69.23%. The numbers indicate that many ineffective fault localization instances are identified correctly, while only very few effective ones are identified wrongly.

Improving fault localization for Simulink models using search-based testing and prediction models

“…Le and Lo [84] propose an approach to predict fault localization accuracy based on features extracted from statistical rankings generated by a fixed and specific test suite. Our predictor model instead is built based on features that compare statistical rankings generated by a test suite and its extensions.…”

“…Moreover, our predictor model is used to help stop test generation and to ensure test suite minimality. Further investigation is required to assess the effectiveness of the features proposed in [84] as a test generation stopping criterion.…”

Theory and Practice, Do They Match? A Case with Spectrum-Based Fault Localization

“…Cheng et al mine graphbased signatures that highlight suspicious program elements by analyzing program spectra [4]. Duy and Lo propose a classification-based approach that predicts whether an SBFL technique would be effective for a particular fault localization task [11]. Xie et al theoretically analyze many SBFL formulas including Tarantula and Ochiai and show that two families of SBFL formulas (ER1 and ER5) could outperform the others if a number of assumptions hold [18].…”

Learning to Combine Multiple Ranking Metrics for Fault Localization

“…Xuan & Monperrus [32] develop test case purification, which seperates test cases into small fractions to enhance the test oracle for fault localization. Le & Lo [13] consider the applicability of fault localization and develop an automatic method for predicting effectiveness with features in faulty programs.…”