Implications of ceiling effects in defect predictors

Abstract

Context: There are many methods that input static code features and output a predictor for faulty code modules. These data mining methods have hit a "performance ceiling"; i.e., some inherent upper bound on the amount of information offered by, say, static code features when identifying modules which contain faults.Objective: We seek an explanation for this ceiling effect. Perhaps static code features have "limited information content"; i.e. their information can be quickly and completely discovered by even simple learners.Method: An initial literature review documents the ceiling effect in other work. Next, using three sub-sampling techniques (under-, over-, and micro-sampling), we look for the lower useful bound on the number of training instances.Results: Using micro-sampling, we find that as few as 50 instances yield as much information as larger training sets.Conclusions: We have found much evidence for the limited information hypothesis. Further progress in learning defect predictors may not come from better algorithms. Rather, we need to be improving the information content of the training data, perhaps with case-based reasoning methods.

Influence of confirmation biases of developers on software quality: an empirical study

“…In other words, the number of defective files is far less than the number of defect-free files. Therefore, we use the under-sampling method, which is the most suitable sampling method for our datasets [16]. The pseudocode of the prediction model is given in Figure 3.…”

“…As a result, both increasing the efficiency of the software testing phase and delivering the software product to the market on time become possible. Reported results in software defect prediction literature suggest that further progress in defect prediction performance can be achieved by increasing the content of input data that defect predictors learn rather than using different algorithms or increasing the size of input data [17], [15], [16]. We can group some significant work in the literature in terms of their focus: algorithm driven approaches; data size driven approaches; and data content driven approaches.…”

“…al. [16] performed a series of experiments. They used Naïve Bayes as their algorithm, since it was useful in their previous experiments [15] as well as J4.8 which was used in prior under-over sampling experiments [18], [19].…”

Predicting Fault-Prone Software Modules with Rank Sum Classification

“…The question is whether there remains additional information in the data set that might be exploited to improve performance. While this is still an open question, and one to which we make some contribution through the present study, there is a view that a performance ceiling has been reached, and that the way forward lies in enriching the data with new information beyond existing metrics [12]. Nevertheless, the NASA data sets are freely available and remain attractive targets for researchers.…”

Active Learning for Software Defect Prediction

“…Although unsupervised learning has been applied, they also have unstable performance [8], [9]. Different from unsupervised learning, the active learning reduces the number of labeled instances required to achieve a stable performance in the majority of reported results [10].…”