The impact of using biased performance metrics on software defect prediction research

Yao, Jingxiu; Shepperd, Martin

doi:10.1016/j.infsof.2021.106664

Cited by 38 publications

(33 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The experimental results show that MKEL method is better against total six methods under F‐1, including coding‐based ensemble learning (CEL) (Sun et al, 2012), dynamic AdaBoost.NC (S. Wang & Yao, 2013), weighted Naïve Bayes (T. Wang & Li, 2010), Compressed C4.5 decision tree (Wang et al, 2012), Cost‐sensitive Boosting Neural Network (CBNN) (Zheng, 2010), and Asymmetric Kernel Principal Component Classification (AKPCC) (Y. Ma et al, 2012). As F‐1 is problematic in defect prediction (Yao & Shepperd, 2021), the experimental results are sceptical. Most of these classical methods are associated with Boosting, but the evaluation indicators of these methods are not unified, which cannot provide a meaningful reference for industry personnel.…”

Section: Related Workmentioning

confidence: 99%

“…Firstly, the class imbalance can reduce the accuracy and reliability of the defect model in CVDP and IVDP, as shown in the result analysis. And, class imbalances can distort some performance indicators about machine learning, for example, accuracy, precision, and F‐measure (Yao & Shepperd, 2021). Based on practical experience, we employ two basic and three comprehensive indicators to evaluate the performance of defect models, improving the reliability of the evaluation results in CVDP and IVDP.…”

Section: Threats To Validitymentioning

confidence: 99%

Section: Construct Validitymentioning

confidence: 99%

“…Ma et al, 2012). As F-1 is problematic in defect prediction (Yao & Shepperd, 2021), the experimental results are sceptical. Most of these classical methods are associated with Boosting, but the evaluation indicators of these methods are not unified, which cannot provide a meaningful reference for industry personnel.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Eliminating the high false‐positive rate in defect prediction through BayesNet with adjustable weight

et al. 2022

View full text Add to dashboard Cite

In defect prediction, a high false‐positive rate (FPR) caused by class imbalance not only increases the workload of testing and development but also consumes unnecessary costs. Many defect models against class imbalance have been proposed to improve the accuracy of defect prediction, but their ability to reduce FPR is unclear. To solve these problems, we first proposed a BayesNet with adjustable weights, called WBN, to reduce the FPR in software defect prediction, which is an algorithm independent of data preprocessing techniques. The mechanism of our WBN is to change the sampling probability of the misclassified instances when training the defect model, making the BayesNet model focus more on false alarm instances. And then, we investigate the FPR of five mainstream defect models for solving class imbalance and select them as comparison models to test the validity of our methods. The experimental result on eight open‐source projects shows that a) our WBN, in in‐version defect prediction (IVDP) and cross‐version defect prediction (CVDP), effectively reduces FPR with means of 0.384 and 0.322, respectively; b) compared with improved subclass discriminant analysis (ISDA) that is the lowest FPR in all control models, our WBN not only reduced the FPR but maintained recall whose mean value was 0.797, whereas ISDA did not, with an average recall of only 0.397; c) our WBN, in CVDP, not only reduces FPR, but also has significant superiority over five control defect models and baseline. Besides, we also found that the class imbalance difference between the test set and the training set has an impact on CVDP performance, recommending that practitioners choose the best dataset for CVDP from the defect data of the historical version through special technology.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Threats To Validitymentioning

confidence: 99%

Section: Construct Validitymentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Eliminating the high false‐positive rate in defect prediction through BayesNet with adjustable weight

et al. 2022

View full text Add to dashboard Cite

show abstract

“…The unsupervised ML technique is applied when working on unlabelled datasets, unlike supervised ML [14]. However, the performance of an ML technique depends largely on the quality of the datasets used for training such an ML technique [47][48][49].…”

Section: Related Workmentioning

confidence: 99%

An Adaptive Rank Aggregation-Based Ensemble Multi-Filter Feature Selection Method in Software Defect Prediction

Balogun

Basri

Capretz

et al. 2021

Entropy

View full text Add to dashboard Cite

Feature selection is known to be an applicable solution to address the problem of high dimensionality in software defect prediction (SDP). However, choosing an appropriate filter feature selection (FFS) method that will generate and guarantee optimal features in SDP is an open research issue, known as the filter rank selection problem. As a solution, the combination of multiple filter methods can alleviate the filter rank selection problem. In this study, a novel adaptive rank aggregation-based ensemble multi-filter feature selection (AREMFFS) method is proposed to resolve high dimensionality and filter rank selection problems in SDP. Specifically, the proposed AREMFFS method is based on assessing and combining the strengths of individual FFS methods by aggregating multiple rank lists in the generation and subsequent selection of top-ranked features to be used in the SDP process. The efficacy of the proposed AREMFFS method is evaluated with decision tree (DT) and naïve Bayes (NB) models on defect datasets from different repositories with diverse defect granularities. Findings from the experimental results indicated the superiority of AREMFFS over other baseline FFS methods that were evaluated, existing rank aggregation based multi-filter FS methods, and variants of AREMFFS as developed in this study. That is, the proposed AREMFFS method not only had a superior effect on prediction performances of SDP models but also outperformed baseline FS methods and existing rank aggregation based multi-filter FS methods. Therefore, this study recommends the combination of multiple FFS methods to utilize the strength of respective FFS methods and take advantage of filter–filter relationships in selecting optimal features for SDP processes.

show abstract

A research landscape on software defect prediction

Taskeen

Khan

Felix

2023

J Software Evolu Process

View full text Add to dashboard Cite

Software defect prediction is the process of identifying defective files and modules that need rigorous testing. In the literature, several secondary studies including systematic reviews, mapping studies, and review studies have been reported. However, no research work such as a tertiary study that combines secondary studies has focused on providing a landscape of software defect prediction useful to understand the body of knowledge. Motivated by this, we intend to perform a tertiary study by following a systematic literature review protocol to provide a research landscape of the targeted domain. We synthesize the quality of the secondary studies and investigate the employed techniques and the performance evaluation measures for evaluating the software defect prediction model. Furthermore, this study aims at exploring different datasets employed in the reported experimentation. Moreover, the current study intends at highlighting the research trends, gaps, and opportunities in the targeted research domain. The results indicate that none of the reported defect prediction techniques can be regarded as the best; however, the reported techniques performed better in different testing situations. In addition, machine learning (ML)‐based techniques perform better than traditional statistical techniques mainly due to the potential of discovering the defects and generating generalized results. Moreover, the obtained results highlight the need for further work in the domain of ML‐based techniques. Furthermore, publicly available datasets should be considered for experimentation or replication purposes. The potential future work can focus on data quality, ethical ML, cross‐project defect prediction, early defect prediction process, class imbalance problem, and model overfitting.

show abstract

The impact of using biased performance metrics on software defect prediction research

Cited by 38 publications

References 62 publications

Eliminating the high false‐positive rate in defect prediction through BayesNet with adjustable weight

Eliminating the high false‐positive rate in defect prediction through BayesNet with adjustable weight

An Adaptive Rank Aggregation-Based Ensemble Multi-Filter Feature Selection Method in Software Defect Prediction

A research landscape on software defect prediction

Contact Info

Product

Resources

About