Stable rankings for different effort models

Menzies, Tim; Jalali, Omid; Hihn, Jairus; Baker, D. R.; Lum, Karen

doi:10.1007/s10515-010-0070-z

Cited by 40 publications

(35 citation statements)

References 41 publications

(48 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recall the results of Shepperd et al [25] and Menzies et al [26]: different experimental conditions can change the rank of an effort estimator. Hence, it is important to study not just the rank of an estimator, but also how well that method performs across multiple experimental conditions such as: As to comparison summaries, the following procedure was repeated for each error measure.…”

Section: Experimental Conditionssupporting

confidence: 53%

See 1 more Smart Citation

On the Value of Ensemble Effort Estimation

Kocagüneli

Menzies

Keung

2012

IIEEE Trans. Software Eng.

Self Cite

226

210

View full text Add to dashboard Cite

Abstract-Background: Despite decades of research, there is no consensus on which software effort estimation methods produce the most accurate models. Aim: Prior work has reported that, given M estimation methods, no single method consistently outperforms all others. Perhaps rather than recommending one estimation method as best, it is wiser to generate estimates from ensembles of multiple estimation methods. Method: 9 learners were combined with 10 pre-processing options to generate 9 × 10 = 90 solo-methods. These were applied to 20 data sets and evaluated using 7 error measures. This identified the best n (in our case n = 13) solo-methods that showed stable performance across multiple datasets and error measures. The top 2, 4, 8 and 13 solo-methods were then combined to generate 12 multi-methods, which were then compared to the solo-methods. Results: (i) The top 10 (out of 12) multi-methods significantly out-performed all 90 solo-methods. (ii) The error rates of the multimethods were significantly less than the solo-methods. (iii) The ranking of the best multi-method was remarkably stable. Conclusion: While there is no best single effort estimation method, there exist best combinations of such effort estimation methods.

show abstract

Section: Experimental Conditionssupporting

confidence: 53%

“…Recently, researchers have access to more methods than that used by Shepperd et al For example, Menzies et al [26] studied 158 methods. While that study was over a very limited data set (just two old COCOMO data sets), their preliminary results prompted this study (where we work with 20 data sets).…”

Section: Ranking Instabilitymentioning

confidence: 99%

On the Value of Ensemble Effort Estimation

Kocagüneli

Menzies

Keung

2012

IIEEE Trans. Software Eng.

Self Cite

226

210

View full text Add to dashboard Cite

show abstract

“…Following the suggestions by Shepperd and Kadoda [15], and Menzies et al [13], we use a large set of data (9 datasets) in our experiment, and use more robust evaluation criteria (5 error measures) [4] subject to Wilcoxon rank-sum statistical test. This evaluation procedure is aimed at providing a stable conclusion in this comparison-based study.…”

Section: Discussionmentioning

confidence: 99%

“…The results from many empirical studies [1,2,9,10] endorsed the effectiveness of dynamic-k over fixed-k methods, but some of which produced conflicting results [10]. This therefore introduces the unstable conclusion, an issue widely recognized in software effort estimation where different studies in the same area provide greatly diversified conclusions [6,13].…”

Section: Introductionmentioning

confidence: 99%

A review and comparison of methods for determining the best analogies in analogy-based software effort estimation

Chinthanet

Phannachitta

Kamei

et al. 2016

Proceedings of the 31st Annual ACM Symposium on Applied Computing

View full text Add to dashboard Cite

Analogy-based effort estimation (ABE) is a commonly used software development effort estimation method. The processes of ABE are based on a reuse of effort values from similar past projects, where the appropriate numbers of past projects (k values) to be reused is one of the long-standing debates in ABE research studies. To date, many approaches to find this k value have been continually proposed. One important reason for this inconclusive debate is that different studies appear to produce different conclusions of the k value to be appropriate. Therefore, in this study, we revisit 8 common approaches to the k value being most appropriate in general situations. With a more robust and comprehensive evaluation methodology using 5 robust error measures subject to the Wilcoxon rank-sum statistical test, we found that conflicting results in the previous studies were not mainly due to the use of different methodologies nor different datasets, but the performance of the different approaches are actually varied widely.

show abstract

“…and we used a fixed, average fixing time value for each defect. Omitting the severity measure in defect prediction studies is a frequent practice [23], however, it can be important when simulated QA cost calculation will be compared to real-life 27 Cost Effectiveness of Software Defect ... Figure 5: Costs of quality assurance in considered project: actual compared to simulated, when DePress-based defect prediction is used values. In our simulation we assumed equal severity for each defect, which is reflected in an equal, average cost (see Table 4).…”

Section: Threats To Validitymentioning

confidence: 99%

Cost Effectiveness of Software Defect Prediction in an Industrial Project

Hryszko

Madeyski

2018

Foundations of Computing and Decision Sciences

View full text Add to dashboard Cite

Abstract. Software defect prediction is a promising approach aiming to increase software quality and, as a result, development pace. Unfortunately, the cost effectiveness of software defect prediction in industrial settings is not eagerly shared by the pioneering companies. In particular, this is the first attempt to investigate the cost effectiveness of using the DePress open source software measurement framework (jointly developed by Wroclaw University of Science and Technology, and Capgemini software development company) for defect prediction in commercial software projects. We explore whether defect prediction can positively impact an industrial software development project by generating profits. To meet this goal, we conducted a defect prediction and simulated potential quality assurance costs based on the best possible prediction results when using a default, non-tweaked DePress configuration, as well as the proposed Quality Assurance (QA) strategy. Results of our investigation are optimistic: we estimated that quality assurance costs can be reduced by almost 30% when the proposed approach will be used, while estimated DePress usage Return on Investment (ROI) is fully 73 (7300%), and Benefits Cost Ratio (BCR) is 74. Such promising results, being the outcome of the presented research, have caused the acceptance of continued usage of the DePress-based software defect prediction for actual industrial projects run by Volvo Group.

show abstract

Stable rankings for different effort models

Cited by 40 publications

References 41 publications

On the Value of Ensemble Effort Estimation

On the Value of Ensemble Effort Estimation

A review and comparison of methods for determining the best analogies in analogy-based software effort estimation

Cost Effectiveness of Software Defect Prediction in an Industrial Project

Contact Info

Product

Resources

About