On the Value of Ensemble Effort Estimation

Abstract

Abstract-Background: Despite decades of research, there is no consensus on which software effort estimation methods produce the most accurate models. Aim: Prior work has reported that, given M estimation methods, no single method consistently outperforms all others. Perhaps rather than recommending one estimation method as best, it is wiser to generate estimates from ensembles of multiple estimation methods. Method: 9 learners were combined with 10 pre-processing options to generate 9 × 10 = 90 solo-methods. These were applied to 20 data sets and evaluated using 7 error measures. This identified the best n (in our case n = 13) solo-methods that showed stable performance across multiple datasets and error measures. The top 2, 4, 8 and 13 solo-methods were then combined to generate 12 multi-methods, which were then compared to the solo-methods. Results: (i) The top 10 (out of 12) multi-methods significantly out-performed all 90 solo-methods. (ii) The error rates of the multimethods were significantly less than the solo-methods. (iii) The ranking of the best multi-method was remarkably stable. Conclusion: While there is no best single effort estimation method, there exist best combinations of such effort estimation methods.

Using improved analytical programming algorithm for effort estimation in software engineering

“…The recent work of Silhavy et al [7] suggest a new approach " automatic complexity estimation based on requirements ", which is partly based on Use Case Points method. Very promising way is a research of Kocaguneli et al [8], this paper shows, that ensemble of effort estimation methods could provide better results than a single estimator.…”

LSA-X: Exploiting Productivity Factors in Linear Size Adaptation for Analogy-Based Software Effort Estimation

“…To avoid the issue of unstable results and to improve generalization of the results, we adopted 4 feature selection methods in this study, all of which are often adopted in software effort estimation literature [4], [13]:…”

“…We select ABE over other alternative model-based methods to study productivity factors because (1) it is widely used, widely studied, and consistently ranked among the bestperforming methods in terms of accuracy [1], [5], [11], [13], and (2) various approaches to improve its accuracy by exploiting productivity measures have been proposed as ways to refine the estimates in the solution adaptation stage. Using ABE to estimate the software development effort involves a 4-stage case-based reasoning process [14], consisting of: Retrieve the project cases most similar to the new case, Reuse the information from the retrieved past cases to propose a solution to the new case, Revise the proposed solution to better adapt to the new case, and Retain the solved case for future problem solving.…”

“…All of these are available in the tera-PROMISE software engineering repository (Accessible through [18]), where the datasets are made available and commonly used in empirical software engineering studies. To maximize the validity of our study, these 19 datasets were taken from the list of 20 datasets which is the highest number of datasets that appeared in software development effort estimation literature [4], [13]. We decided to exclude one dataset, namely Telecom, because it does not contain any software size variables, which are required by the LSA, MSA, RTM and the proposed LSA-X techniques.…”

Which models of the past are relevant to the present? A software effort estimation approach to exploiting useful past models

“…More recent work has been emphasising the relatively good predictive performance achieved by ensembles of learning machines (Kultur et al 2009;Minku and Yao 2013a;Kocaguneli et al 2012) and local methods that make estimations based on completed projects similar to the project being estimated (Minku and Yao 2013a;Menzies et al 2013;Bettenburg et al 2012). For instance, Regression Trees (RTs), Bagging ensembles of MultiLayer Perceptrons (Bag + MLPs) and Bagging ensembles of RTs (Bag + RTs) have been shown to perform well across several datasets (Minku and Yao 2013a).…”