Revisiting the evaluation of defect prediction models

Abstract

Defect Prediction Models aim at identifying error-prone parts of a software system as early as possible. Many such models have been proposed, their evaluation, however, is still an open question, as recent publications show.An important aspect often ignored during evaluation is the effort reduction gained by using such models. Models are usually evaluated per module by performance measures used in information retrieval, such as recall, precision, or the area under the ROC curve (AUC). These measures assume that the costs associated with additional quality assurance activities are the same for each module, which is not reasonable in practice. For example, costs for unit testing and code reviews are roughly proportional to the size of a module.In this paper, we investigate this discrepancy using optimal and trivial models. We describe a trivial model that takes only the module size measured in lines of code into account, and compare it to five classification methods. The trivial model performs surprisingly well when evaluated using AUC. However, when an effort-sensitive performance measure is used, it becomes apparent that the trivial model is in fact the worst.

Untitled

“…Traditional performance metrics used in most previous work are precision, recall, f-measure, AUC [23], error sum, median error, error variance, and correlation [3]. Mende and Koschke [6], Arisholm et al [25], and Rahman et al [26] suggested that traditional performance metrics are not wellsuited for evaluating defect prediction approaches in a practical scenario. Indeed, for traditional metrics, all defect prone software artifacts have the same priority while software engineers would benefit from identifying those software components containing more defects earlier.…”

“…Indeed, for traditional metrics, all defect prone software artifacts have the same priority while software engineers would benefit from identifying those software components containing more defects earlier. As pointed out by Mende and Koschke [6] and D'Ambros et al [3] the scenario that is more useful in practice is to rank the classes by the predicted number of defects they will exhibit. In the context of defect prediction, prediction models assign a defect probability to all classes, according to which they can be ranked.…”

“…Thus, the main goal of defect prediction approaches is to maximize the percentage of defects that can be encountered when reviewing only the top k% of classes [3]. This can be shown visually via a cumulative lift chart [28], [29], where the classes are ordered according to the prediction model on the x axis while the cumulative number of actual defects is plotted on the y axis. Similarly, in the context of change prediction, Koru et al [30] proposed the usage of regression tree to predict the number of future changes in two releases of two software projects.…”

“…More recent work on prediction models raised the issue that the effort that developers have to put into inspecting artifacts suggested by binary classification models varies depending on the artifact [6]. Larger and more complex software artifacts require additional inspection effort, which might limit both the usefulness and the effectiveness of the prediction, especially when the prediction is used as a means to better allocate resources.…”

“…In other words, prediction models should be effort-aware and try to rank software artifacts based on the changes/defects they will exhibit, placing those with most changes/defects at the top. In this case, commonly used evaluation metrics are: the Spearman's rank correlation between predicted and actual ranking, and the P opt metric [6], defined as the difference between the AUC of the optimal ranking (considering the inspection of n% of artifacts) and the AUC of the prediction model.…”