A simulation study of the model evaluation criterion mmre

Foss, Tron; Stensrud, Erik; Kitchenham, Barbara; Myrtveit, Ingunn

doi:10.1109/tse.2003.1245300

Cited by 390 publications

(292 citation statements)

References 40 publications

Supporting

Mentioning

278

Contrasting

Unclassified

Order By: Relevance

“…Thus, this paper still uses the above criteria. However, in addition to that, this paper uses the absolute residual measures because it has shown that the absolute residual measures, in particular the SD Ab.Res., are a better measure than MRE for model comparison [18].…”

Section: Prediction Accuracy Measuresmentioning

confidence: 99%

See 1 more Smart Citation

An application of Bayesian network for predicting object-oriented software maintainability

Koten

Gray

2006

Information and Software Technology

154

100

View full text Add to dashboard Cite

The Department of Information Science is one of six departments that make up the School of Business at the University of Otago. The department offers courses of study leading to a major in Information Science within the BCom, BA and BSc degrees. In addition to undergraduate teaching, the department is also strongly involved in postgraduate research programmes leading to MCom, MA, MSc and PhD degrees. Research projects in spatial information processing, connectionist-based information systems, software engineering and software development, information engineering and database, software metrics, distributed information systems, multimedia information systems and information systems security are particularly well supported.The views expressed in this paper are not necessarily those of the department as a whole. The accuracy of the information presented in this paper is the sole responsibility of the authors. CopyrightCopyright remains with the authors. Permission to copy for research or teaching purposes is granted on the condition that the authors and the Series are given due acknowledgment. Reproduction in any form for purposes other than research or teaching is forbidden unless prior written permission has been obtained from the authors. CorrespondenceThis paper represents work to date and may not necessarily form the basis for the authors' final conclusions relating to this topic. It is likely, however, that the paper will appear in some form in a journal or in conference proceedings in the near future. The authors would be pleased to receive correspondence in connection with any of the issues raised in this paper, or for subsequent publication details. Please write directly to the authors at the address provided below. (Details of final journal/conference publication venues for these papers are also provided on the Department's publications web pages: http://www.otago.ac.nz/informationscience/pubs/). Any other correspondence concerning the Series should be sent to the DPS Coordinator. AbstractAs the number of object-oriented software systems increases, it becomes more important for organizations to maintain those systems effectively. However, currently only a small number of maintainability prediction models are available for objectoriented systems. This paper presents a Bayesian network maintainability prediction model for an object-oriented software system. The model is constructed using object-oriented metric data in Li and Henry's datasets, which were collected from two different object-oriented systems. Prediction accuracy of the model is evaluated and compared with commonly used regression-based models. The results suggest that the Bayesian network model can predict maintainability more accurately than the regression-based models for one system, and almost as accurately as the best regression-based model for the other system.

show abstract

Section: Prediction Accuracy Measuresmentioning

confidence: 99%

“…On the other hand, there is a concern about MRE because MRE is biased [33] and not always reliable as a prediction accuracy measure [18]. However, MRE has been the de facto standard in the software effort prediction literature and no alternative standard exists at present.…”

Section: Prediction Accuracy Measuresmentioning

confidence: 99%

An application of Bayesian network for predicting object-oriented software maintainability

Koten

Gray

2006

Information and Software Technology

154

100

View full text Add to dashboard Cite

show abstract

“…Table 3 summarizes the 5 error measures we used in our experiments. These 5 error measures were selected according to the suggestion by a study of Foss et al [26], in which the validity of numerous error measures commonly used in the software development effort estimation literature were systematically evaluated. These 5 measures are considerably more robust and valid than many others such as the well-known MMRE and Pred (25).…”

Section: Evaluation Methodologymentioning

confidence: 99%

“…Software effort estimation studies are often criticized for inadequate use of evaluation metrics such as error measures. For example, use of MMRE by itself is commonly seen in the literature [26], despite widespread criticism of MMRE as an inappropriate and biased measure. However, in this study, we maximized the construct validity by using 5 robust error measures, all of which were guaranteed there robustness by a statistical method [26].…”

Section: Threats To Validitymentioning

confidence: 99%

LSA-X: Exploiting Productivity Factors in Linear Size Adaptation for Analogy-Based Software Effort Estimation

Phannachitta

Monden

Keung

et al. 2016

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARYAnalogy-based software effort estimation has gained a considerable amount of attention in current research and practice. Its excellent estimation accuracy relies on its solution adaptation stage, where an effort estimate is produced from similar past projects. This study proposes a solution adaptation technique named LSA-X that introduces an approach to exploit the potential of productivity factors, i.e., project variables with a high correlation with software productivity, in the solution adaptation stage. The LSA-X technique tailors the exploitation of the productivity factors with a procedure based on the Linear Size Adaptation (LSA) technique. The results, based on 19 datasets show that in circumstances where a dataset exhibits a high correlation coefficient between productivity and a related factor (r ≥ 0.30), the proposed LSA-X technique statistically outperformed (95% confidence) the other 8 commonly used techniques compared in this study. In other circumstances, our results suggest using any linear adaptation technique based on software size to compensate for the limitations of the LSA-X technique.

show abstract

“…They use the NASA KC1 data set to evaluate their approach, but unfortunately their only performance measure was Magnitude of Relative Error. In literature, this metric has been believed to be asymmetric [27,50], and the use of supporting measures is usually desired to avoid any doubts of validity. Additionally, when they apply their prediction method to an external data set (data set not used in training), their MMRE reaches scores of up to 159%, with a maximum MRE of about 373%.…”

Section: Defect-correction Effort Predictionmentioning

confidence: 99%

An effort prediction framework for software defect correction

Hassouna¹,

Tahvildari²

2010

Information and Software Technology

View full text Add to dashboard Cite

Developers apply changes and updates to software systems to adapt to emerging environments and address new requirements. In turn, these changes introduce additional software defects, usually caused by our inability to comprehend the full scope of the modified code. As a result, software practitioners have developed tools to aid in the detection and prediction of imminent software defects, in addition to the effort required to correct them. Although software development effort prediction has been in use for many years, research into defect-correction effort prediction is relatively new. The increasing complexity, integration and ubiquitous nature of current software systems has sparked renewed interest in this field. Effort prediction now plays a critical role in the planning activities of managers. Accurate predictions help corporations budget, plan and distribute available resources effectively and efficiently. In particular, early defect-correction effort predictions could be used by testers to set schedules, and by managers to plan costs and provide earlier feedback to customers about future releases.In this work, we address the problem of predicting the effort needed to resolve a software defect. More specifically, our study is concerned with defects or issues that are reported on an Issue Tracking System or any other defect repository. Current approaches use one prediction method or technique to produce effort predictions. This approach usually suffers from the weaknesses of the chosen prediction method, and consequently the accuracy of the predictions are affected. To address this problem, we present a composite prediction framework. Rather than using one prediction approach for all defects, we propose the use of multiple integrated methods which complement the weaknesses of one another. Our framework is divided into two sub-categories, Similarity-Score Dependent and Similarity-Score Independent. The Similarity-Score Dependent method utilizes the power of Case-Based Reasoning, also known as Instance-Based Reasoning, to compute predictions. It relies on matching target issues to similar historical cases, then combines their known effort for an informed estimate. On the other hand, the Similarity-Score Independent method makes use of other defect-related information with some statistical manipulation to produce the required estimate. To measure similarity between defects, some method of distance calculation must be used. In some cases, this method might produce misleading results due to observed inconsistencies in history, and the fact that current similarity-scoring techniques cannot account for all the variability in the data. In this case, the Similarity-Score Independent method can be used to estimate the effort, where the effect of such inconsistencies can be reduced.We have performed a number of experimental studies on the proposed framework to assess the effectiveness of the presented techniques. We extracted the data sets from an operational Issue Tracking System in order to test the validity...

show abstract

A simulation study of the model evaluation criterion mmre

Abstract: The Index termsMean magnitude of relative error, simulation, regression analysis, prediction models, software engineering.

Cited by 390 publications

References 40 publications

An application of Bayesian network for predicting object-oriented software maintainability

An application of Bayesian network for predicting object-oriented software maintainability

LSA-X: Exploiting Productivity Factors in Linear Size Adaptation for Analogy-Based Software Effort Estimation

An effort prediction framework for software defect correction

Contact Info

Product

Resources

About