ELBlocker: Predicting blocking bugs with ensemble imbalance learning

Xia, Xin; Lo, David; Shihab, Emad; Wang, Xinyu; Yang, Xiaohu

doi:10.1016/j.infsof.2014.12.006

Cited by 90 publications

(54 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, the SMOTE class rebalancing technique has a configurable parameter that need to be specified. Similar to prior studies in software engineering [3,69,92], the results of our study rely on one default parameter setting (i.e., k = 5).…”

Section: Construct Validitymentioning

confidence: 95%

The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models

Tantithamthavorn

Hassan

Matsumoto

2020

IIEEE Trans. Software Eng.

234

142

View full text Add to dashboard Cite

Defect prediction models that are trained on class imbalanced datasets (i.e., the proportion of defective and clean modules is not equally represented) are highly susceptible to produce inaccurate prediction models. Prior research compares the impact of class rebalancing techniques on the performance of defect prediction models. Prior research efforts arrive at contradictory conclusions due to the use of different choice of datasets, classification techniques, and performance measures. Such contradictory conclusions make it hard to derive practical guidelines for whether class rebalancing techniques should be applied in the context of defect prediction models. In this paper, we investigate the impact of 4 popularly-used class rebalancing techniques on 10 commonly-used performance measures and the interpretation of defect prediction models. We also construct statistical models to better understand in which experimental design settings that class rebalancing techniques are beneficial for defect prediction models. Through a case study of 101 datasets that span across proprietary and open-source systems, we recommend that class rebalancing techniques are necessary when quality assurance teams wish to increase the completeness of identifying software defects (i.e., Recall). However, class rebalancing techniques should be avoided when interpreting defect prediction models. We also find that class rebalancing techniques do not impact the AUC measure. Hence, AUC should be used as a standard measure when comparing defect prediction models.

show abstract

Section: Construct Validitymentioning

confidence: 95%

The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models

Tantithamthavorn

Hassan

Matsumoto

2020

IIEEE Trans. Software Eng.

234

142

View full text Add to dashboard Cite

show abstract

“…Threats to construct validity refer to the suitability of our evaluation metrics. We use cost effectiveness and F-measure which are also used by past software engineering studies to evaluate the effectiveness of various prediction techniques [26], [27], [12], [15], [28], [27], [29]. Thus, we believe there is little threat to construct validity.…”

Section: Threats To Validitymentioning

confidence: 99%

An Empirical Study of Classifier Combination for Cross-Project Defect Prediction

Zhang

Xia

et al. 2015

2015 IEEE 39th Annual Computer Software and Applications Conference

Self Cite

View full text Add to dashboard Cite

Abstract-To help developers better allocate testing and debugging efforts, many software defect prediction techniques have been proposed in the literature. These techniques can be used to predict classes that are more likely to be buggy based on past history of buggy classes. These techniques work well as long as a sufficient amount of data is available to train a prediction model. However, there is rarely enough training data for new software projects. To deal with this problem, cross-project defect prediction, which transfers a prediction model trained using data from one project to another, has been proposed and is regarded as a new challenge for defect prediction. So far, only a few cross-project defect prediction techniques have been proposed. To advance the state-of-the-art, in this work, we investigate 7 composite algorithms, which integrate multiple machine learning classifiers, to improve cross-project defect prediction. To evaluate the performance of the composite algorithms, we perform experiments on 10 open source software systems from the PROMISE repository which contain a total of 5,305 instances labeled as defective or clean. We compare the composite algorithms with CODEPLogistic, which is the latest cross-project defect prediction algorithm proposed by Panichella et al. [1], in terms of two standard evaluation metrics: cost effectiveness and F-measure. Our experiment results show that several algorithms outperform CODEPLogistic: Max performs the best in terms of F-measure and its average F-measure outperforms that of CODEPLogistic by 36.88%. BaggingJ48 performs the best in terms of cost effectiveness and its average cost effectiveness outperforms that of CODEPLogistic by 15.34%.

show abstract

“…In this work, we are interested in identifying build cochanges, and follow the prior work [10], [16], [17], we use random forest [15] to construct a classifier. In the model building phase, random forest constructs a number of decision trees by using instances in the training set.…”

Section: A Imbalanced Classifiermentioning

confidence: 99%

Cross-project build co-change prediction

Xia

McIntosh

et al. 2015

2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER)

Self Cite

View full text Add to dashboard Cite

Build systems orchestrate how human-readable source code is translated into executable programs. In a software project, source code changes can induce changes in the build system (aka. build co-changes). It is difficult for developers to identify when build co-changes are necessary due to the complexity of build systems. Prediction of build co-changes works well if there is a sufficient amount of training data to build a model. However, in practice, for new projects, there exists a limited number of changes. Using training data from other projects to predict the build co-changes in a new project can help improve the performance of the build co-change prediction. We refer to this problem as cross-project build co-change prediction.In this paper, we propose CroBuild, a novel cross-project build co-change prediction approach that iteratively learns new classifiers. CroBuild constructs an ensemble of classifiers by iteratively building classifiers and assigning them weights according to its prediction error rate. Given that only a small proportion of code changes are build co-changing, we also propose an imbalanceaware approach that learns a threshold boundary between those code changes that are build co-changing and those that are not in order to construct classifiers in each iteration. To examine the benefits of CroBuild, we perform experiments on 4 large datasets including Mozilla, Eclipse-core, Lucene, and Jazz, comprising a total of 50,884 changes. On average, across the 4 datasets, CroBuild achieves a F1-score of up to 0.408. We also compare CroBuild with other approaches such as a basic model, AdaBoost proposed by Freund et al., and TrAdaBoost proposed by Dai et al.. On average, across the 4 datasets, the CroBuild approach yields an improvement in F1-scores of 41.54%, 36.63%, and 36.97% over the basic model, AdaBoost, and TrAdaBoost, respectively.

show abstract

ELBlocker: Predicting blocking bugs with ensemble imbalance learning

Cited by 90 publications

References 39 publications

The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models

The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models

An Empirical Study of Classifier Combination for Cross-Project Defect Prediction

Cross-project build co-change prediction

Contact Info

Product

Resources

About