A systematic mapping study on cross-project defect prediction

Herbold, Steffen

doi:10.48550/arxiv.1705.06429

Cited by 5 publications

(7 citation statements)

References 77 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The independent variables of our study are performance metrics of defect prediction experiments. The selection of these metrics is based on two systematic literature studies on defect prediction research (Hosseini et al 2019;Herbold 2017), with the goal to cover a broad set of metrics that are currently used. Overall, we identified 20 performance metrics.…”

Section: Independent Variablesmentioning

confidence: 99%

“…Thus, while all these studies consider different angles, the conclusions are similar: performance metrics are often problematic because their absolute values are unreliable proxies for the practical relevance of the prediction performance and the relation between performance metrics and cost savings is rather indirect. This is perhaps also the reason why there is no consensus in the defect prediction community regarding the question which metrics should be used, which leads to a large number of different criteria that are used (Hosseini et al 2019;Herbold 2017) and has potentially severe effects on the validity of our work like researcher bias (Shepperd et al 2014;Tantithamthavorn et al 2016b;Shepperd et al 2018).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Exploring the relationship between performance metrics and cost saving potential of defect prediction models

Herbold

2022

Empir Software Eng

Self Cite

View full text Add to dashboard Cite

Context: Performance metrics are a core component of the evaluation of any machine learning model and used to compare models and estimate their usefulness. Recent work started to question the validity of many performance metrics for this purpose in the context of software defect prediction. Objective: Within this study, we explore the relationship between performance metrics and the cost saving potential of defect prediction models. We study whether performance metrics are suitable proxies to evaluate the cost saving capabilities and derive a theory for the relationship between performance metrics and cost saving potential. Methods: We measure performance metrics and cost saving potential in defect prediction experiments. We use a multinomial logit model, decision, and random forest to model the relationship between the metrics and the cost savings. Results: We could not find a stable relationship between cost savings and performance metrics. We attribute the lack of the relationship to the inability of performance metrics to account for the property that a small proportion of very large software artifacts are the main driver of the costs. fact that performance metrics are incapable of accurately considering the costs associate with individual artifacts, which is required due to the exponential distribution of artifact sizes. Conclusion: Any defect prediction study interested in finding the best prediction model, must consider cost savings directly, because no reasonable claims regarding the economic benefits of defect prediction can be made otherwise.

show abstract

Section: Independent Variablesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Exploring the relationship between performance metrics and cost saving potential of defect prediction models

Herbold

2022

Empir Software Eng

Self Cite

View full text Add to dashboard Cite

show abstract

“…One of the biggest concerns when carrying out experiments using open-source artifacts is related to the representativeness of the artifacts and whether therefore the results found can be properly generalized [Hillenbrand and Gayvert, 2005].…”

Section: Overview Of Projectsmentioning

confidence: 99%

“…The effectiveness of fault prediction techniques is often demonstrated using historical repository data [Herbold, 2017]. However, once these techniques are adopted, it is not clear how they would affect projects that do not match with the characteristics (language, platform, domain) of the built model [He et al, 2015].…”

Section: Analyzing Fault Pronenessmentioning

confidence: 99%

Towards the Systematic Testing of Virtual Reality Programs

Andrade

Nunes

Delamaro

2019

2019 21st Symposium on Virtual and Augmented Reality (SVR)

View full text Add to dashboard Cite

Software testing is a critical activity to ensure that software complies with its specification. However, current software testing activities tend not to be completely effective when applied in specific software domains in Virtual Reality (VR) that has several new types of features such as images, sounds, videos, and differentiated interaction, which can become sources of new kinds of faults. This paper presents an overview of the main VR characteristics that can have an impact on verification, validation, and testing (VV&T). Furthermore, it analyzes some of the most successful VR open-source projects to draw a picture concerning the danger of the lack of software testing activities. We compared the current state of software testing practice in open-source VR projects and evaluate how the lack of testing can be damaging to the development of a product. We assessed the incidence of code smells and verified how such projects behave concerning the tendency to present faults. We also perform the same analyses on projects that are not VR related to have a better understanding of these results. The results showed that the practice of software testing is not yet widespread in the development of VR applications. It was also found that there is a high incidence of code smells in VR projects. Analyzing Non-VR projects we noticed that classes that have test cases tend to produce fewer smells compared to classes that were not tested. Regarding fault-proneness analysis, we used an unsupervised approach to VR and Non-VR projects. Results showed that about 12.2% of the classes analyzed in VR projects are fault-prone, while Non-VR projects presented a lower fault-proneness rate (8.9%). Regarding the application of software testing techniques on VR projects, it was observed that only a small number of projects are concerned about developing test cases for VR projects, perhaps because we still do not have the necessary tools to help in this direction. Concerning smells, we concluded that there is a high incidence in VR projects, especially regarding implementing smells and this high incidence can have a significant influence on faults. Finally, the study related to fault proneness pointed out that the lack of software testing activity is a significant risk to the success of the projects.

show abstract

“…Many CPDP work mainly focus on verifying the feasibility of the CPDP methods [11,24,28,33,47]. Briand et al [6] firstly propose cross project defect prediction, using the open source soware Xpose defect data to make predictions for Jwriter.…”

Section: Related Workmentioning

confidence: 99%

An empirical study of public data quality problems in cross project defect prediction

Sun,

Li,

Sun

2018

Preprint

View full text Add to dashboard Cite

Background: Two public defect data, including Jureczko and NASA datasets, have been widely used in cross project defect prediction (CPDP). e quality of defect data have been reported as an important factor influencing the defect prediction performance and Shepperd et al. have researched the data quality problems in NASA datasets. However, up to now, there is no research focusing on the quality problems of Jureczko datasets which are most widely used in CPDP. Aims: In this paper, we intend to investigate the problems of identical and inconsistent cases in Jureczko datasets and validate whether removing these problematic cases will make a difference to defect prediction performance in CPDP. Method: e problems of identical and inconsistent cases are reported from two aspects, respectively in each individual dataset and in a pair of datasets from different releases of a so ware project. en a cleaned version of Jureczko datasets is provided by removing duplicate and inconsistent cases. Finally three training data selection methods are employed to compare the defect prediction performance of cleaned datasets with that of original datasets. Results: e experimental results in terms of AUC and F-Measure show that most datasets obtain very different defect prediction performance. Conclusions: It is very necessary to study the data quality problems in CPDP and the cleaned Jureczko datasets may provide more reliable defect prediction performance in CPDP.

show abstract

A systematic mapping study on cross-project defect prediction

Abstract: The term defect is used throughout this paper. In the literature, the terms fault and bug are also used interchangeably.

Cited by 5 publications

References 77 publications

Exploring the relationship between performance metrics and cost saving potential of defect prediction models

Exploring the relationship between performance metrics and cost saving potential of defect prediction models

Towards the Systematic Testing of Virtual Reality Programs

An empirical study of public data quality problems in cross project defect prediction

Contact Info

Product

Resources

About