An Empirical Study of Training Data Selection Methods for Ranking-Oriented Cross-Project Defect Prediction

Abstract

Ranking-oriented cross-project defect prediction (ROCPDP), which ranks software modules of a new target industrial project based on the predicted defect number or density, has been suggested in the literature. A major concern of ROCPDP is the distribution difference between the source project (aka. within-project) data and target project (aka. cross-project) data, which evidently degrades prediction performance. To investigate the impacts of training data selection methods on the performances of ROCPDP models, we examined the practical effects of nine training data selection methods, including a global filter, which does not filter out any cross-project data. Additionally, the prediction performances of ROCPDP models trained on the filtered cross-project data using the training data selection methods were compared with those of ranking-oriented within-project defect prediction (ROWPDP) models trained on sufficient and limited within-project data. Eleven available defect datasets from the industrial projects were considered and evaluated using two ranking performance measures, i.e., FPA and Norm(Popt). The results showed no statistically significant differences among these nine training data selection methods in terms of FPA and Norm(Popt). The performances of ROCPDP models trained on filtered cross-project data were not comparable with those of ROWPDP models trained on sufficient historical within-project data. However, ROCPDP models trained on filtered cross-project data achieved better performance values than ROWPDP models trained on limited historical within-project data. Therefore, we recommended that software quality teams exploit other project datasets to perform ROCPDP when there is no or limited within-project data.

The impact of training data selection on the software defect prediction performance and data complexity

“…This approach does not require a labeled target dataset, which an organization may not have when building an SDP model. The representative training data selection methods were chosen based on their frequent use in SDP comparative studies [22]- [24], the granularity of the training data, and the selection strategy. Table 5 lists the methods.…”

“…The improvement in the defect prediction performance is small [22]. Studies [23], [24] also found that the SDP model constructed from selected training data underperforms the baseline model. It encourages us to investigate the factors that may affect the efficacy of the training data selection method.…”

“…It encourages us to investigate the factors that may affect the efficacy of the training data selection method. We used data complexity measures to investigate the selected training dataset characteristics to understand how the findings in [22]- [24] could have occurred. No dedicated study has examined the impact of selecting training data on the data complexity in software defect prediction.…”

A recommendation system of training data selection method for cross-project defect prediction

“…Likewise, this approach does not require a labeled target data set, which an organization may not have when building an SDP model. The representative methods used in this study were chosen based on several factors, including their frequent use in SDP comparative studies [11], [13], [21], the granularity of the training data, and the selection strategy. Instance-level Clustering instances in the same cluster as target instance.…”