Perceptions, Expectations, and Challenges in Defect Prediction

Wan, Zhiyuan; Xia, Xin; Hassan, Ahmed E.; Lo, David; Yin, Jianwei; Yang, Xiaohu

doi:10.1109/tse.2018.2877678

Cited by 128 publications

(65 citation statements)

References 89 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…7. Note that we have confirmed with one of the authors of the survey study [87] that top k%LOC Recall is one of the top-5 measures, not top k%LOC Precision as reported in the paper.…”

Section: Evaluation Measuressupporting

confidence: 84%

“…Defect models at various granularity levels have been proposed, e.g., packages [43], components [82], modules [44], files [43,53], methods [28]. However, developers could still waste an SQA effort on manually identifying the most risky lines, since the current prediction granularity is still perceived as coarse-grained [87]. Hence, the linelevel defect prediction should be beneficial to SQA teams to spend optimal effort on identifying and analyzing defects.…”

Section: The Granularity Levels Of Defect Predictions Modelsmentioning

confidence: 99%

“…To evaluate the approaches, we use five performance measures preferred by practitioners [87], i.e., recall, false alarm rate, a combination of recall and false alarm rate, initial false alarm, and Top k%LOC Recall. 7 In addition, we use Matthews Correlation Coefficients (MCC) to evaluate the overall predictive accuracy which is suitable for the unbalanced data like our line-level defect datasets [8,71].…”

Section: Evaluation Measuresmentioning

confidence: 99%

“…To achieve effective SQA prioritization, defect prediction models have been long investigated at different granularity levels, for example, packages [43], components [82], modules [44], files [43,53], methods [28], and commits [45]. However, developers could still waste an SQA effort on manually identifying the most risky lines, since the current prediction granularity is still perceived as coarse-grained [87]. In addition, our motivating analysis shows that as little as 1%-3% of the lines of code in a file are actually defective after release, suggesting that developers could waste their SQA effort on up to 99% of clean lines of a defective file.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Predicting Defective Lines Using a Model-Agnostic Technique

Wattanakriengkrai

Thongtanunam

Tantithamthavorn

et al. 2022

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Defect prediction models are proposed to help a team prioritize source code areas files that need Software Quality Assurance (SQA) based on the likelihood of having defects. However, developers may waste their unnecessary effort on the whole file while only a small fraction of its source code lines are defective. Indeed, we find that as little as 1%-3% of lines of a file are defective. Hence, in this work, we propose a novel framework (called LINE-DP) to identify defective lines using a model-agnostic technique, i.e., an Explainable AI technique that provides information why the model makes such a prediction. Broadly speaking, our LINE-DP first builds a file-level defect model using code token features. Then, our LINE-DP uses a state-of-the-art model-agnostic technique (i.e., LIME) to identify risky tokens, i.e., code tokens that lead the file-level defect model to predict that the file will be defective. Then, the lines that contain risky tokens are predicted as defective lines. Through a case study of 32 releases of nine Java open source systems, our evaluation results show that our LINE-DP achieves an average recall of 0.61, a false alarm rate of 0.47, a top 20%LOC recall of 0.27, and an initial false alarm of 16, which are statistically better than six baseline approaches. Our evaluation shows that our LINE-DP requires an average computation time of 10 seconds including model construction and defective identification time. In addition, we find that 63% of defective lines that can be identified by our LINE-DP are related to common defects (e.g., argument change, condition change). These results suggest that our LINE-DP can effectively identify defective lines that contain common defects while requiring a smaller amount of inspection effort and a manageable computation cost. The contribution of this paper builds an important step towards line-level defect prediction by leveraging a model-agnostic technique.

show abstract

“…7. Note that we have confirmed with one of the authors of the survey study [87] that top k%LOC Recall is one of the top-5 measures, not top k%LOC Precision as reported in the paper.…”

Section: Evaluation Measuressupporting

confidence: 84%

Section: The Granularity Levels Of Defect Predictions Modelsmentioning

confidence: 99%

Section: Evaluation Measuresmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Predicting Defective Lines Using a Model-Agnostic Technique

Wattanakriengkrai

Thongtanunam

Tantithamthavorn

et al. 2022

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…Software defect prediction (SDP) [18], [25], [46] can construct models by mining version control systems and bug tracking systems, and then uses the constructed models to predict defective modules in advance. Therefore, limited software quality assurance (SQA) resources can be reasonably allocated on these identified defective modules, which can effectively improve the quality of deployed software.…”

Section: Introductionmentioning

confidence: 99%

ALTRA: Cross-Project Software Defect Prediction via Active Learning and Tradaboost

et al. 2020

View full text Add to dashboard Cite

Cross-project defect prediction (CPDP) methods can be used when the target project is a new project or lacks enough labeled program modules. In these new target projects, we can easily extract and then measure these modules with software measurement tools. However, labeling these program modules is time-consuming, error-prone and requires professional domain knowledge. Moreover, directly using labeled modules in the other projects (i.e., the source projects) can not achieve satisfactory performance due to the large data distribution difference in most cases. In this article, to our best knowledge, we are the first to propose a novel method ALTRA, which can utilize both active learning and TrAdaBoost to alleviate this issue. In particular, we firstly use Burak filter to select similar labeled modules from the source project after analyzing the unlabeled modules in the target project. Then we use active learning to choose representative unlabeled modules from the target project and ask experts to label the type (i.e., defective or non-defective) of these modules. Later, we use TrAdaBoost to determine the weights of labeled modules in the source project and the target project, and then construct the model via weighted support vector machine. After selecting a small number of modules (i.e., only 5% modules) in the target project, we terminate the method ALTRA and return the final constructed model. To show the effectiveness of our proposed method ALTRA, we choose 10 large-scale open-source projects from different application domains. In terms of both F1 and AUC performance indicators, we find ALTRA can perform significantly better than seven state-of-the-art CPDP baselines. Moreover, we also show that the usage of Burak filter, the uncertainty active learning strategy, the class imbalanced learning method and TrAdaBoost are competitive in our proposed method ALTRA.

show abstract

Predictive maintenance of infrastructure code using “fluid” datasets: An exploratory study on Ansible defect proneness

Quattrocchi

Tamburri

2022

J Software Evolu Process

View full text Add to dashboard Cite

This work consolidates and compounds previous investigations in recognizing defects for infrastructure‐as‐code (IaC) scripts using general software development quality metrics with a focus on defect severity but adding to previous work an explorative look at creating datasets, which may boost the predictive power of provided models—we call this notion a fluid dataset. More specifically, we experiment with 50 different metrics harnessing a multiple dataset creation process whereby different versions of the same datasets are rigged with auto‐training facilities for model retraining and redeployment in a DataOps fashion. At this point, with a focus on the Ansible infrastructure code language—as a de facto standard for industrial‐strength infrastructure code—we build defect prediction models and manage to improve on the state of the art by finding an F1 score of 0.52 and a recall of 0.57 using a Naive–Bayes classifier. On the one hand, by improving state‐of‐the‐art defect prediction models using metrics generalizable for different IaC languages, we provide interesting leads for the future of infrastructure‐as‐code. On the other hand, we have barely scratched the surface on the novel approach of fluid‐datasets creation and automated retraining of Machine Learning (ML) defect prediction models, warranting for more research on the same direction in the future.

show abstract

Perceptions, Expectations, and Challenges in Defect Prediction

Cited by 128 publications

References 89 publications

Predicting Defective Lines Using a Model-Agnostic Technique

Predicting Defective Lines Using a Model-Agnostic Technique

ALTRA: Cross-Project Software Defect Prediction via Active Learning and Tradaboost

Predictive maintenance of infrastructure code using “fluid” datasets: An exploratory study on Ansible defect proneness

Contact Info

Product

Resources

About