What accuracy statistics really measure

Kitchenham, Barbara; Pickard, Lesley M.; MacDonell, Stephen G.; Shepperd, Martin

doi:10.1049/ip-sen:20010506

Cited by 361 publications

(284 citation statements)

References 4 publications

Supporting

Mentioning

274

Contrasting

Unclassified

Order By: Relevance

“…The most significant improvement is in AAR, where we get an improvement of about 8 Hours. (25) and 10% for PRED (50). The overall FEEDBACK is lower than that of the Base Approach (as expected and explained in Section 4.2.1 of Chapter 4).…”

Section: Enhancement Evaluationsupporting

confidence: 69%

“…α-kNN shows a noticeable improvement as α increases. Starting at α = 0.1, with AAR ≈ 14 Hours, PRED(25) ≈ 15% and PRED(50) ≈ 25%, we get approximately a 15% improvement for PRED (25) and a 20% improvement for PRED (50). The most significant improvement is in AAR, where we get an improvement of about 8 Hours.…”

Section: Enhancement Evaluationmentioning

confidence: 85%

“…They use the NASA KC1 data set to evaluate their approach, but unfortunately their only performance measure was Magnitude of Relative Error. In literature, this metric has been believed to be asymmetric [27,50], and the use of supporting measures is usually desired to avoid any doubts of validity. Additionally, when they apply their prediction method to an external data set (data set not used in training), their MMRE reaches scores of up to 159%, with a maximum MRE of about 373%.…”

Section: Defect-correction Effort Predictionmentioning

confidence: 99%

“…In addition, we use multiple performance metrics to assess the accuracy of the framework. Most performance metrics have some kind of bias [50], therefore we opt to using complimentary metrics that show the distribution of error for our predictions in a more comprehensive way.…”

Section: Defect-correction Effort Predictionmentioning

confidence: 99%

“…However, it is closely related to the Sum of Absolute Residuals, which considered an unbiased statistic [50].…”

Section: Performance Metricsmentioning

confidence: 99%

See 4 more Smart Citations

An effort prediction framework for software defect correction

Hassouna¹,

Tahvildari²

2010

Information and Software Technology

View full text Add to dashboard Cite

Developers apply changes and updates to software systems to adapt to emerging environments and address new requirements. In turn, these changes introduce additional software defects, usually caused by our inability to comprehend the full scope of the modified code. As a result, software practitioners have developed tools to aid in the detection and prediction of imminent software defects, in addition to the effort required to correct them. Although software development effort prediction has been in use for many years, research into defect-correction effort prediction is relatively new. The increasing complexity, integration and ubiquitous nature of current software systems has sparked renewed interest in this field. Effort prediction now plays a critical role in the planning activities of managers. Accurate predictions help corporations budget, plan and distribute available resources effectively and efficiently. In particular, early defect-correction effort predictions could be used by testers to set schedules, and by managers to plan costs and provide earlier feedback to customers about future releases.In this work, we address the problem of predicting the effort needed to resolve a software defect. More specifically, our study is concerned with defects or issues that are reported on an Issue Tracking System or any other defect repository. Current approaches use one prediction method or technique to produce effort predictions. This approach usually suffers from the weaknesses of the chosen prediction method, and consequently the accuracy of the predictions are affected. To address this problem, we present a composite prediction framework. Rather than using one prediction approach for all defects, we propose the use of multiple integrated methods which complement the weaknesses of one another. Our framework is divided into two sub-categories, Similarity-Score Dependent and Similarity-Score Independent. The Similarity-Score Dependent method utilizes the power of Case-Based Reasoning, also known as Instance-Based Reasoning, to compute predictions. It relies on matching target issues to similar historical cases, then combines their known effort for an informed estimate. On the other hand, the Similarity-Score Independent method makes use of other defect-related information with some statistical manipulation to produce the required estimate. To measure similarity between defects, some method of distance calculation must be used. In some cases, this method might produce misleading results due to observed inconsistencies in history, and the fact that current similarity-scoring techniques cannot account for all the variability in the data. In this case, the Similarity-Score Independent method can be used to estimate the effort, where the effect of such inconsistencies can be reduced.We have performed a number of experimental studies on the proposed framework to assess the effectiveness of the presented techniques. We extracted the data sets from an operational Issue Tracking System in order to test the validity...

show abstract

Section: Enhancement Evaluationsupporting

confidence: 69%

Section: Enhancement Evaluationmentioning

confidence: 85%

Section: Defect-correction Effort Predictionmentioning

confidence: 99%

Section: Defect-correction Effort Predictionmentioning

confidence: 99%

“…However, it is closely related to the Sum of Absolute Residuals, which considered an unbiased statistic [50].…”

Section: Performance Metricsmentioning

confidence: 99%

See 3 more Smart Citations

An effort prediction framework for software defect correction

Hassouna¹,

Tahvildari²

2010

Information and Software Technology

View full text Add to dashboard Cite

show abstract

Measures of Model Performance Based On the Log Accuracy Ratio

2018

View full text Add to dashboard Cite

Quantitative assessment of modeling and forecasting of continuous quantities uses a variety of approaches. We review existing literature describing metrics for forecast accuracy and bias, concentrating on those based on relative errors and percentage errors. Of these accuracy metrics, the mean absolute percentage error (MAPE) is one of the most common across many fields and has been widely applied in recent space science literature and we highlight the benefits and drawbacks of MAPE and proposed alternatives. We then introduce the log accuracy ratio and derive from it two metrics: the median symmetric accuracy and the symmetric signed percentage bias. Robust methods for estimating the spread of a multiplicative linear model using the log accuracy ratio are also presented. The developed metrics are shown to be easy to interpret, robust, and to mitigate the key drawbacks of their more widely used counterparts based on relative errors and percentage errors. Their use is illustrated with radiation belt electron flux modeling examples.

show abstract

A novel adaptive structure for SOA system effort estimation

Mishra

Kumar

2016

Trans. Emerging Tel. Tech.

View full text Add to dashboard Cite

This paper addresses a novel adaptive structure to solve the problem of service‐oriented architecture (SOA) system effort estimation. The objective of our work is to combine the insights of signals theory with empirical research to find a recommendation scheme for the problem of SOA system effort estimation. The main motivation for using this structure is to enhance the principle of self‐adaptability to the situation at hand. The proposed structure consists of an adaptive filter composed of linear combiner of filter weights, input and desired values. Additionally, a gradient steepest descent method is used to adjust the filter parameters (training process) using the least mean square algorithm. Furthermore, an experimental analysis is conducted with the proposed structure using the data of 10 past SOA system industrial applications, and in the empirical analysis, two performance measurement metrics and an evaluation function are used to assess the performance in terms of predictive accuracy. The experimental analysis and comparison study are helpful to demonstrate the effectiveness of an estimation technique. The obtained results indicate that an improved predictive accuracy for the problem of SOA system effort estimation has been achieved using the proposed structure when compared with support vector machines, linear, stepwise and ordinary least square regression techniques. Copyright © 2016 John Wiley & Sons, Ltd.

show abstract

What accuracy statistics really measure

Abstract: Abstract

Cited by 361 publications

References 4 publications

An effort prediction framework for software defect correction

An effort prediction framework for software defect correction

Measures of Model Performance Based On the Log Accuracy Ratio

A novel adaptive structure for SOA system effort estimation

Contact Info

Product

Resources

About