A principled evaluation of ensembles of learning machines for software effort estimation

Abstract

Background: Software effort estimation (SEE) is a task of strategic importance in software management. Recently, some studies have attempted to use ensembles of learning machines for this task. Aims: We aim at (1) evaluating whether readily available ensemble methods generally improve SEE given by single learning machines and which of them would be more useful; getting insight on (2) how to improve SEE; and (3) how to choose machine learning (ML) models for SEE. Method: A principled and comprehensive statistical comparison of three ensemble methods and three single learners was carried out using thirteen data sets. Feature selection and ensemble diversity analyses were performed to gain insight on how to improve SEE based on the approaches singled out. In addition, a risk analysis was performed to investigate the robustness to outliers. Therefore, the better understanding/insight provided by the paper is based on principled experiments, not just an intuition or speculation. Results: None of the compared methods is consistently the best, even though regression trees and bagging using multilayer perceptrons (MLPs) are more frequently among the best. These two approaches usually perform similarly. Regression trees place more important features in higher levels of the trees, suggesting that feature weights are important when using ML models for SEE. The analysis of bagging with MLPs suggests that a self-tuning ensemble diversity method may help improving SEE. Conclusions: Ideally, principled experiments should be done in an individual basis to choose a model. If an organisation has no resources for that, regression trees seem to be a good choice for its simplicity. The analysis also suggests approaches to improve SEE.

Software effort estimation as a multiobjective learning problem

“…Chulani et al [1999] presented another study using a Bayesian approach to combine a priori information based on expert knowledge to a linear regression model based on log transformation of the data. The proposed approach outperforms linear regression models based on log transformed [Minku and Yao 2011]. data in terms of PRED(20), PRED(25) and PRED(30) on one data set and its extended version. No comparison with Shepperd and Schofield [1997]'s work was given.…”

“…In the present work, we look into a type of machine learning method which has recently attracted attention from the SEE community [Braga et al 2007;Kultur et al 2009;Kocaguneli et al 2009;Minku and Yao 2011], namely ensembles of learning machines. Ensembles are sets of learners trained to perform the same task and combined with the aim of improving predictive performance [Chen and Yao 2009].…”

“…Simply using any general purpose ensemble approach from the literature does not necessarily improve SEE in comparison to single learning machines [Minku and Yao 2011]. Minku and Yao [2013] showed that combining the power of ensembles to local learning through the use of bagging ensembles of regression trees outperforms several other learning machines for SEE in terms of Mean Absolute Error (MAE).…”

Which models of the past are relevant to the present? A software effort estimation approach to exploiting useful past models

“…The parameters of the RTs were minimum total weight of one for the instances in a leaf, and minimum proportion 0.0001 of the variance on all the data that need to be present at a node in order for splitting to be performed. These parameters were shown to be appropriate in the literature (Minku and Yao 2011).…”

Cross-validation based K nearest neighbor imputation for software quality datasets: An empirical study

“…Less attention has been focused on MDT methods themselves. In a more recent study, Huang et al (2015) found that only some of the former software effort estimation studies have considered the significance of the MDTs, of which only Minku and Yao (2011) (Myrtveit et al, 2001;Strike et al, 2001), and the prediction error may be introduced (Mittas and Angelis, 2010). MEI is efficient and has been involved in SEE as the most popular imputation approach; however, it will cause bias to data.…”