ERDeR: The Combination of Statistical Shrinkage Methods and Ensemble Approaches to Improve the Performance of Deep Regression

Abstract

Ensembling is a powerful technique to obtain the most accurate results. In some cases, the large number of learners in ensemble learning mostly increases both computational load during the test phase and error rate. To solve this problem, in this paper we propose an Ensemble of Reduced Deep Regression (ERDeR) model, which is a combination of Deep Regressions (DRs), shrinkage methods, and ensemble approaches. The framework of the proposed model contains three phases. The first phase includes base regressions in which parallel DRs are used as learners. The role of these DRs is to extract features of input data and make prediction. In the second phase, to automatically reduce and select the most suitable DRs, shrinkage methods such as Least Absolute Shrinkage and Selection Operator (LASSO) and Elastic Net (EN) are employed. These models are compared with the non-shrinkage model. The last phase is ensemble phase, which consists of three different ensemble methods namely Multi-Layer Perceptron (MLP), Weighted Average (WA), and Simple Average (SA). These ensemble methods are used to aggregate the remaining learners from previous steps. Finally, the proposed model is applied to Monte Carlo simulation data and three real datasets including Boston House Price, Real Estate Valuation and Gold Price per Ounce. The results show that after applying the shrinkage methods the error rate is significantly reduced and the model accuracy is increased. Accordingly, the results of combining shrinkage methods and ensemble approaches not only decreased the computational load during test phase, but also increased the model accuracy.