On the relative value of cross-company and within-company data for defect prediction

Abstract

We propose a practical defect prediction approach for companies that do not track defect related data. Specifically, we investigate the applicability of cross-company (CC) data for building localized defect predictors using static code features.Firstly, we analyze the conditions, where CC data can be used as is. These conditions turn out to be quite few. Then we apply principles of analogy-based learning (i.e. nearest neighbor (NN) filtering) to CC data, in order to fine tune these models for localization. We compare the performance of these models with that of defect predictors learned from within-company (WC) data. As expected, we observe that defect predictors learned from WC data outperform the ones learned from CC data. However, our analyses also yield defect predictors learned from NN-filtered CC data, with performance close to, but still not better than, WC data. Therefore, we perform a final analysis for determining the minimum number of local defect reports in order to learn WC defect predictors. We demonstrate in this paper that the minimum number of data samples required to build effective defect predictors can be quite small and can be collected quickly within a few months.Hence, for companies with no local defect data, we recommend a two-phase approach that allows them to employ the defect prediction process instantaneously. In phase one, companies should use NN-filtered CC data to initiate the defect prediction process and simultaneously start collecting WC (local) data. Once enough WC data is collected (i.e. after a few months), organizations should switch to phase two and use predictors learned from WC data.

Cost Effectiveness of Software Defect Prediction in an Industrial Project

“…Two other publications of Khoshgoftaar and Seliya from 2004 [15] and 2005 [16] continued with the previous concept and focused on commercial data analysis, but were not applied to a real-world environment. A similar approach can be found in publications by Ostrand and Weyuker [29], Ostrand et al [31], Tosun et al [45], Turhan et al [47,48]. Examples of industrial applications of information gathered by using defect prediction can be found in publications by Wong et al [50], Succi et al [41] and Kläs et al [17].…”

A Data Filtering Method Based on Agglomerative Clustering

“…The performance of CCDP is generally poor because of larger irrelevant CC data. Previous work [23] found that using raw CC data directly would increase false alarm rates due to irrelevant instance in CC data, so several data filtering works should be done before building the prediction model. For example, Turhan et al [23] and Peters et al [24] proposed the NN filter and the Perters filter to select the CC instances which are mostly similar to WC data as the training dataset.…”

“…The first one is to apply the data filtering method to find the best suitable training data (e.g., [23,24,26]). For example, Turhan et al [23] proposed a nearest neighbor (NN) filter to select cross-company data.…”

“…For example, Turhan et al [23] proposed a nearest neighbor (NN) filter to select cross-company data. Peters et al [24] introduced the Peters filter to select training data via the structure of other projects.…”

Cross-Cultural Personality Prediction based on Twitter Data

“…We also conducted some preliminary experiments to create a personality prediction model for a country. While there are many software engineering papers which utilize from cross-company or cross-project data such as cross-project defect prediction [25]- [31] and cross-company effort estimation [32]- [38], there is no cross-cultural personality prediction study yet. Therefore, we see a big research potential on the analysis of cross-cultural personality prediction models.…”