Effort-Aware Defect Prediction Models

Mende, Thilo; Koschke, Rainer

doi:10.1109/csmr.2010.18

Cited by 174 publications

(100 citation statements)

References 31 publications

Supporting

Mentioning

100

Contrasting

Order By: Relevance

“…R dd (x) is defined as #errors(x) / E(x), where #errors(x) is the number of faults in module x and E(x) is the required effort for module x. We use LOC for E(x) as previous studies [22] [23]. In this case, R dd (x) means fault density.…”

Section: E Effort-aware Evaluation (Rq1) 1) Effort-aware Model and Pmentioning

confidence: 99%

“…If AUC is around 0.5 or less, the prediction is meaningless. "ddr" is the "defect detection rate" [22]. In this paper, "ddr x " means fault detection rate at the x% of effort.…”

Section: E Effort-aware Evaluation (Rq1) 1) Effort-aware Model and Pmentioning

confidence: 99%

“…To involve this discussion, we use the effort-aware model proposed by Mende et al [22]. In the effort-aware model, the relative risk R dd (x) is used to prioritize modules to test or inspect.…”

Section: E Effort-aware Evaluation (Rq1) 1) Effort-aware Model and Pmentioning

confidence: 99%

“… For RQ1, we evaluate prediction performances using binary classification in subsection D and using faultdensity prediction and effort-aware models [22] in subsection E.…”

Section: Rq2: Does Adding Impactscale To Existing Product Metrics Andmentioning

confidence: 99%

See 3 more Smart Citations

ImpactScale: Quantifying change impact to predict faults in large software systems

Kobayashi

Matsuo

Inoue

et al. 2011

2011 27th IEEE International Conference on Software Maintenance (ICSM)

View full text Add to dashboard Cite

Abstract-In software maintenance, both product metrics and process metrics are required to predict faults effectively. However, process metrics cannot be always collected in practical situations. To enable accurate fault prediction without process metrics, we define a new metric, ImpactScale. ImpactScale is the quantified value of change impact, and the change propagation model for ImpactScale is characterized by probabilistic propagation and relation-sensitive propagation. To evaluate ImpactScale, we predicted faults in two large enterprise systems using the effort-aware models and Poisson regression. The results showed that adding ImpactScale to existing product metrics increased the number of detected faults at 10% effort (LOC) by over 50%. ImpactScale also improved the predicting model using existing product metrics and dependency network measures.

show abstract

Section: E Effort-aware Evaluation (Rq1) 1) Effort-aware Model and Pmentioning

confidence: 99%

“…If AUC is around 0.5 or less, the prediction is meaningless. "ddr" is the "defect detection rate" [22]. In this paper, "ddr x " means fault detection rate at the x% of effort.…”

Section: E Effort-aware Evaluation (Rq1) 1) Effort-aware Model and Pmentioning

confidence: 99%

Section: E Effort-aware Evaluation (Rq1) 1) Effort-aware Model and Pmentioning

confidence: 99%

“… For RQ1, we evaluate prediction performances using binary classification in subsection D and using faultdensity prediction and effort-aware models [22] in subsection E.…”

Section: Rq2: Does Adding Impactscale To Existing Product Metrics Andmentioning

confidence: 99%

See 2 more Smart Citations

ImpactScale: Quantifying change impact to predict faults in large software systems

Kobayashi

Matsuo

Inoue

et al. 2011

2011 27th IEEE International Conference on Software Maintenance (ICSM)

View full text Add to dashboard Cite

show abstract

“…It is a similar case for 2 more of the 8 data sets used in (Menzies, Greenwald & Frank 2007), where n is the approximate average number of 'defective' predictions made. This is known as LOC module-order modelling (see (Khoshgoftaar & Allen 2003) and (Mende & Koschke 2009)), and highlights both the poor predictive performance of the classifiers, and that research into making defect predictors 'effort aware' is worthwhile (see (Arisholm, Briand & Fuglerud 2007) and (Mende & Koschke 2010)). Table II presents statistics for each of the 13 NASA Metrics Data Program data sets.…”

Section: Why Class Distribution Affects the Suitability Of Measuresmentioning

confidence: 99%

Further thoughts on precision

Gray

Bowes

Davey

et al. 2011

15th Annual Conference on Evaluation &Amp; Assessment in Software Engineering (EASE 2011)

View full text Add to dashboard Cite

Abstract-Background: There has been much discussion amongst automated software defect prediction researchers regarding use of the precision and false positive rate classifier performance metrics. Aim: To demonstrate and explain why failing to report precision when using data with highly imbalanced class distributions may provide an overly optimistic view of classifier performance. Method: Well documented examples of how dependent class distribution affects the suitability of performance measures. Conclusions: When using data where the minority class represents less than around 5 to 10 percent of data points in total, failing to report precision may be a critical mistake. Furthermore, deriving the precision values omitted from studies can reveal valuable insight into true classifier performance.

show abstract

Large‐scale inter‐system clone detection using suffix trees and hashing

Koschke

2013

J Software Evolu Process

View full text Add to dashboard Cite

Detecting a similar code between two systems has various applications such as comparing two software variants or versions or finding potential license violations. Techniques detecting suspiciously similar code must scale in terms of resources needed to very large code corpora and need to have high precision because a human needs to inspect the results. This paper demonstrates how suffix trees can be used to obtain a scalable comparison. The evaluation is carried out for very large code corpora. Our evaluation shows that our approach is faster than index-based techniques when the analysis is run only once. If the analysis is to be conducted multiple times, creating an index pays off. We report how much code can be filtered out from the analysis using an index-based filter. In addition to that, this paper proposes a method to improve precision through user feedback. A user validates a sample of the found clone candidates. An automated data mining technique learns a decision tree on the basis of the user decisions and different code metrics. We investigate the relevance of several metrics and whether criteria learned from one application domain can be generalized to other domains.All of the aforementioned variants of clone detection are facing challenges with respect to detection quality and scalability. Detection quality requires high recall and high precision in finding the relevant code. Relevance depends on the use case. In particular, inter-system and intra-system clone detections need to deal with re-occurring similar code that is similar from a lexical or syntactical point of view, but that is not interesting for the given task. Frequent examples of such irrelevant similar code are import statement lists, array initializers, setter/getter sequences, or sequences of pure declarations or simple assignments.Another challenge is scalability. Whereas intra-system clone detection searches only within one system, inter-system clone search may face a much larger code base, often larger by orders of magnitude. Also, fragment search may face this problem, when the code is searched in very large software repositories [3,4].Several researchers have recently proposed to use an index-based code search to address scalability for the search in very large code bases [13,3,4,17,18].The index-based techniques first create an index against which code of a subject system is compared later. The purpose of the index is to identify the code that has a chance of being similar. The code filtered out by the index is not compared. The index is a first seed of a similar code fragment. This seed is then extended by merging with neighboring similar code fragments [13,3,4].Creating the index can be expensive. The idea is to invest upfront in an index that is created only once but whose cost is amortized in multiple subsequent searches.Contributions. Our conference paper introduced a way to extend traditional suffix-tree-based clone detection for inter-system clone search that scales for very large programs [19]. This approach avoids the nee...

show abstract

Effort-Aware Defect Prediction Models

Cited by 174 publications

References 31 publications

ImpactScale: Quantifying change impact to predict faults in large software systems

ImpactScale: Quantifying change impact to predict faults in large software systems

Further thoughts on precision

Large‐scale inter‐system clone detection using suffix trees and hashing

Contact Info

Product

Resources

About