Ensemble effort estimation with metaheuristic hyperparameters and weight optimization for achieving accuracy

Abstract

Software development effort estimation (SDEE) is recognized as vital activity for effective project management since under or over estimating can lead to unsuccessful utilization of project resources. Machine learning (ML) algorithms are largely contributing in SDEE domain, particularly ensemble effort estimation (EEE) works well in rectifying bias and subjectivity to solo ML learners. Performance of EEE significantly depends on hyperparameter composition as well as weight assignment mechanism of solo learners. However, in EEE domain, impact of optimization in terms of hyperparameter tunning as well as weight assignment is explored by few researchers. This study aims in improving SDEE performance by incorporating metaheuristic hyperparameter and weight optimization in EEE, which enables accuracy and diversity to the ensemble model. The study proposed Metaheuristic-optimized Multi-dimensional bagging scheme and Weighted Ensemble (MoMdbWE) approach. This is achieved by proposed search space division and hyperparameter optimization method named as Multi-dimensional bagging (Mdb). Metaheuristic algorithm considered for this work is Firefly algorithm (FFA), to get best hyperparameters of three base ML algorithms (Random Forest, Support vector machine and Deep Neural network) since FFA has shown promising results of fitness in terms of MAE. Further enhancement in performance is achieved by incorporating FFA-based weight optimization to construct Metaheuristic-optimized weighted ensemble (MoWE) of individual multi-dimensional bagging schemes. Proposed scheme is implemented on eight frequently utilized effort estimation datasets and results are evaluated by 5 error metrices (MAE, RMSE, MMRE, MdMRE, Pred), standard accuracy and effect size along with Wilcox statistical test. Findings confirmed that the use of FFA optimization for hyperparameter (with search space sub-division) and for ensemble weights, has significantly enhanced performance in comparison with individual base algorithms as well as other homogeneous and heterogenous EEE techniques.