A framework for comparing multiple cost estimation methods using an automated visualization toolkit

Mittas, Nikolaos; Mamalikidis, Ioannis; Angelis, Lefteris

doi:10.1016/j.infsof.2014.05.010

Cited by 33 publications

(20 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…SA has been used to assess the performance of SDEE techniques in a number of studies. [15][16][17][18] Kocaguneli et al used SA along with 6 accuracy metrics (MBRE, MIBRE, mean magnitude of error relative, MMRE, Pred (0.25), and MAR) to compare cross-company and within-company datasets using a self-tuning analogy-based effort estimation method called TEAK. 18 They found that different accuracy metrics can result in different conclusions.…”

Section: Standardized Accuracymentioning

confidence: 99%

Evaluating Pred(p) and standardized accuracy criteria in software development effort estimation

Idri

Abnane

Abran

2017

J Software Evolu Process

View full text Add to dashboard Cite

Software development effort estimation (SDEE) plays a primary role in software project management. But choosing the appropriate SDEE technique remains elusive for many project managers and researchers. Moreover, the choice of a reliable estimation accuracy measure is crucial because SDEE techniques behave differently given different accuracy measures. The most widely used accuracy measures in SDEE are those based on magnitude of relative error (MRE) such as mean/median MRE (MMRE/MedMRE) and prediction at level p (Pred(p)), which counts the number of observations where an SDEE technique gave MREs lower than p. However, MRE has proven to be an unreliable accuracy measure, favoring SDEE techniques that underestimate. Consequently, an unbiased measure called standardized accuracy (SA) has been proposed. This paper deals with the Pred(p) and SA measures. We investigate (1) the consistency of Pred(p) and SA as accuracy measures and SDEE technique selectors, and (2) the relationship between Pred(p) and SA. The results suggest that Pred(p) is less biased towards underestimates and generally selects the same best technique as SA. Moreover, SA and Pred(p) measure different aspects of technique performance, and SA may be used as a predictor of Pred(p) by means of the 3 association rules.

show abstract

Section: Standardized Accuracymentioning

confidence: 99%

Evaluating Pred(p) and standardized accuracy criteria in software development effort estimation

Idri

Abnane

Abran

2017

J Software Evolu Process

View full text Add to dashboard Cite

show abstract

“…In addition, the ongoing use of a baseline model in the literature would give a single point of comparison, allowing a meaningful assessment of any new method against previous work. Work in SEE involving regression error curves [Bi and Bennett 2003;Mittas and Angelis 2012] and model comparison frameworks [Mittas et al 2015] have included a naïve baseline model prediction using the mean or median of the data. In addition, a baseline model using the mean of a set of random samplings (with replacement) has also been proposed [Shepperd and MacDonell 2012].…”

Section: Baseline Modelsmentioning

confidence: 99%

A Baseline Model for Software Effort Estimation

Whigham

Owen

MacDonell

2015

ACM Trans. Softw. Eng. Methodol.

112

View full text Add to dashboard Cite

Software effort estimation (SEE) is a core activity in all software processes and development lifecycles. A range of increasingly complex methods has been considered in the past 30 years for the prediction of effort, often with mixed and contradictory results. The comparative assessment of effort prediction methods has therefore become a common approach when considering how best to predict effort over a range of project types. Unfortunately, these assessments use a variety of sampling methods and error measurements, making comparison with other work difficult. This article proposes an automatically transformed linear model (ATLM) as a suitable baseline model for comparison against SEE methods. ATLM is simple yet performs well over a range of different project types. In addition, ATLM may be used with mixed numeric and categorical data and requires no parameter tuning. It is also deterministic, meaning that results obtained are amenable to replication. These and other arguments for using ATLM as a baseline model are presented, and a reference implementation described and made available. We suggest that ATLM should be used as a baseline of effort prediction quality for all future model comparisons in SEE. . 2015. A baseline model for software effort estimation.

show abstract

“…22,23,70 In other words, each technique can be ranked differently according to different accuracy criteria, which can lead to contradictory results (ie, 1 criterion selects model A as the best, whereas another selects model B). 94 For example, Idri et al 70 showed that not all the best techniques according to SA were also the best with regard to Pred (p).…”

Section: Methodology Usedmentioning

confidence: 99%

“…The rationale for using many performance measures is that prior studies showed that selection of the best estimation technique depends on which performance indicator was used since relying on only 1 criterion may lead to biased conclusions . In other words, each technique can be ranked differently according to different accuracy criteria, which can lead to contradictory results (ie, 1 criterion selects model A as the best, whereas another selects model B) . For example, Idri et al showed that not all the best techniques according to SA were also the best with regard to Pred (p). Build and evaluate filter fuzzy analogy ensembles (RQ2) …”

Section: Empirical Designmentioning

confidence: 99%

Evaluating filter fuzzy analogy homogenous ensembles for software development effort estimation

Hosni

Idri

Abran

2018

J Software Evolu Process

View full text Add to dashboard Cite

Researchers have developed and evaluated many techniques to deliver accurate estimates of the effort required to complete a new software program. Among these, analogy has emerged as a very promising technique, in particular the fuzzy analogy estimation technique that uses the fuzzy logic concepts in order to deal with both categorical and numerical data. The aim of this paper is twofold: (1) evaluate the impact of 3 filters on the predictive ability of single and ensemble fuzzy analogy techniques and (2) assess whether filters could be a source of diversity for fuzzy analogy ensembles. Moreover, it compares the filter single and ensemble fuzzy analogy techniques with fuzzy analogy ensembles built without using feature selection over 6 datasets. The overall results suggest that (1) more accurate estimates are generated when filters were used with single and ensemble fuzzy analogy techniques, (2) filter single fuzzy analogy techniques outperformed filter fuzzy analogy ensembles, and (3) fuzzy analogy ensembles without feature selection were more accurate than filter single and ensemble techniques. Therefore, though the use of feature selection techniques led single and ensemble fuzzy analogy to generate accurate estimations, they failed to be a source of diversity for fuzzy analogy ensembles. Hence, constructing fuzzy analogy homogenous ensembles that combine single fuzzy analogy techniques with different parameter configurations still generate better accuracy than filter fuzzy analogy ensembles. However, further empirical evaluations of filter/wrappers fuzzy analogy ensembles are required in order to confirm or refute these findings.

show abstract

A framework for comparing multiple cost estimation methods using an automated visualization toolkit

Cited by 33 publications

References 24 publications

Evaluating Pred(p) and standardized accuracy criteria in software development effort estimation

Evaluating Pred(p) and standardized accuracy criteria in software development effort estimation

A Baseline Model for Software Effort Estimation

Evaluating filter fuzzy analogy homogenous ensembles for software development effort estimation

Contact Info

Product

Resources

About