Which process metrics can significantly improve defect prediction models? An empirical study

Abstract

The knowledge about the software metrics which serve as defect indicators is vital for the efficient allocation of resources for quality assurance. It is the process metrics, although sometimes difficult to collect, which have recently become popular with regard to defect prediction. However, in order to identify rightly the process metrics which are actually worth collecting, we need the evidence validating their ability to improve the product metric-based defect prediction models. This paper presents an empirical evaluation in which several process metrics were investigated in order to identify the ones which significantly improve the defect prediction models based on product metrics. Data from a wide range of software projects (both, industrial and open source) were collected. The predictions of the models that use only product metrics (simple models) were compared with the predictions of the models which used product metrics, as well as one of the process metrics under scrutiny (advanced models). To decide whether the improvements were significant or not, statistical tests were performed and effect sizes were calculated. The advanced defect prediction models trained on a data set containing product metrics and additionally Number of Distinct Committers (NDC) were significantly better than the simple models without NDC, while the effect size was medium and the probability of superiority (PS) of the advanced models over simple ones was high (p ¼ :016, r ¼ À:29, PS ¼ :76), which is a substantial finding useful in defect prediction. A similar result with slightly smaller PS was achieved by the advanced models trained on a data set containing product metrics and additionally all of the investigated process metrics (p ¼ :038, r ¼ À:29, PS ¼ :68). The advanced models trained on a data set containing product metrics and additionally Number of Modified Lines (NML) were significantly better than the simple models without NML, but the effect size was small

Would wider adoption of reproducible research be beneficial for empirical software engineering research?

“…We decided to develop the reproducer R package [27] and made it available from CRAN -the official repository of R packages. Data sets analyzed in our three research papers [20,22,28] are encapsulated in the reproducer R package, while most of the figures and tables (particularly those which depend on data), as well as computational results, are built on the fly from data sets stored in the reproducer package and automatically exported into the manuscript rather than copied. -All references were stored in the pure BibTeX format.…”

“…We will use a real data set recently analysed by Madeyski and Jureczko [28] and available from the reproducer R package [27]. The analytic example presented in this section is based on empirical comparison of simple and advanced software defect prediction models performed on thirty-four (15 industrial and 19 open source) versions of software projects.…”

“…Researchers with more experience of RR concepts can view the reproducer R package [27] including data sets analyzed in our recently published research papers [20,22,28].…”

Cost Effectiveness of Software Defect Prediction in an Industrial Project

“…Low investment costs and high recall of even simple defect prediction performed by DePress can result with high N etReturn of DePress-aided quality assurance planned on a basis of the proposed effort allocation strategy. More sophisticated prediction models, especially ones using software process metrics [11,20], may help to achieve even more impressive results. It is also worth mentioning that cross-project software defect prediction [10,12] is sometimes used to reduce costs.…”

Software Defect Prediction Using Radial Basis and Probabilistic Neural Networks

“…Process metrics can be used to improve software development and maintenance such as the effectiveness of defect removal during development, the pattern of testing defects arrival, and the response time of the fix process. Project metrics describe the project characteristics and execution which includes the number of software developers, the staffing pattern over the life cycle of the software, cost, schedule, and productivity [3] [4]. Software defect prediction refers to those models that try to predict potential software defects from test data.…”