A cross-entropy based stacking method in ensemble learning

Abstract

Stacking is one of the major types of ensemble learning techniques in which a set of base classifiers contributes their outputs to the meta-level classifier, and the meta-level classifier combines them so as to produce more accurate classifications. In this paper, we propose a new stacking algorithm that defines the cross-entropy as the loss function for the classification problem. The training process is conducted by using a neural network with the stochastic gradient descent technique. One major characteristic of our method is its treatment of each meta instance as a whole with one optimization model, which is different from some other stacking methods such as stacking with multi-response linear regression and stacking with multi-response model trees. In these methods each meta instance is divided into a set of sub-instances. Multiple models apply to those sub-instances and each for a class label. There is no connection between different models. It is very likely that our treatment is a better choice for finding suitable weights. Experiments with 22 data sets from the UCI machine learning repository show that the proposed stacking approach performs well. It outperforms all three base classifiers, several state-of-the-art stacking algorithms, and some other representative ensemble learning methods on average.

A multimodal fusion enabled ensemble approach for human activity recognition in smart homes

“…It is observable that in almost all the cases an ensemble is more effective than an individual classifier-based approach. 27 In the following we review several ensemble approaches in the domain of HAR. Peng et al 28 proposed a classifier-level fusion method to recognize complex activities by using acceleration, vital sign, and location data.…”

On the Cost-Effectiveness of Stacking of Neural and Non-Neural Methods for Text Classification: Scenarios and Performance Prediction

“…Ensemble methods, such as Stacking, are strategies that combine multiple models to improve the prediction generalization in a given task. Such methods have shown promise in ATC [Džeroski andŽenko 2004, Ding andWu 2020], enjoying high effectiveness and computational costs that depend on the selected learning methods of the ensemble. Among the possible ensemble strategies, Stacking has the characteristic of using a meta-layer capable of combining the prediction outputs of different heterogeneous individual models.…”

“…However, the benefits of ensemble techniques against a robust classifier are not always clear [Yan-Shi Dong and Ke-Song Han 2004], in part, due to the excellent generalization power of the best classifiers. In fact, previous ensemble works primarily focus on improving the overall classification effectiveness using the results of traditional classification algorithms [Campos et al 2017, Ding andWu 2020], paying little or no attention to practical issues such as the execution time or which combination of efficient base algorithms can bring effective results at a lower cost.…”

On the Cost-Effectiveness of Stacking of Neural and Non-Neural Methods for Text Classification: Scenarios and Performance Prediction

“…However, the benefits of ensemble techniques against a strong classifier are not always clear (Yan-Shi Dong and Ke-Song Han, 2004), in part, due to the excellent generalization power of the best classifiers. In fact, previous ensemble works mostly focus on improving the overall classification effectiveness using the results of traditional classification algorithms (Campos et al, 2017;Ding and Wu, 2020), paying little or no attention to practical issues such as the execution time or which combination of efficient base algorithms can bring effective results at a lower cost. Accordingly, our first contribution in this paper is a thorough study of the cost-effectiveness tradeoff of stacking techniques for text classification tasks.…”

“…Ensemble approaches, such as stacking, which combine the outputs of several base classification models to form an integrated output, have also been shown to excel in ATC (Džeroski and Ženko, 2004;Ding and Wu, 2020), enjoying high effectiveness and computational costs that depend on the selected learning methods of the ensemble. They are motivated by the fact that distinct learning models or text representations may complement each other, uncovering specific structures that underlie the input/output relationship of the data.…”