Ensemble learning: A survey

Abstract

Ensemble methods are considered the state-of-the art solution for many machine learning challenges. Such methods improve the predictive performance of a single model by training multiple models and combining their predictions. This paper introduce the concept of ensemble learning, reviews traditional, novel and state-ofthe-art ensemble methods and discusses current challenges and trends in the field. This article is categorized under: Algorithmic Development > Ensemble Methods Technologies > Machine Learning Technologies > Classification K E Y W O R D S boosting, classifier combination, ensemble models, machine-learning, mixtures of experts, multiple classifier system, random forest 1 | INTRODUCTIONEnsemble learning is an umbrella term for methods that combine multiple inducers to make a decision, typically in supervised machine learning tasks. An inducer, also referred as a base-learner, is an algorithm that takes a set of labeled examples as input and produces a model (e.g., a classifier or regressor) that generalizes these examples. By using the produced model, predictions can be drawn for new unlabeled examples. An ensemble inducer can be of any type of machine learning algorithm (e.g., decision tree, neural network, linear regression model, etc.). The main premise of ensemble learning is that by combining multiple models, the errors of a single inducer will likely be compensated by other inducers, and as a result, the overall prediction performance of the ensemble would be better than that of a single inducer. Ensemble learning is usually regarded as the machine learning interpretation for the wisdom of the crowd. This concept can be illustrated through the story of Sir Francis Galton who was an English philosopher and statistician that conceived the basic concept of standard deviation and correlation. While visiting a livestock fair, Galton conducted a simple weight guessing contest. The participants were asked to guess the weight of an ox. Hundreds of people participated in this contest, but no one succeeded in guessing the weight: 1,198 pounds. Much to his surprise, Galton found that the average of all guesses came quite close to the exact weight: 1,198 pounds. In this experiment, Galton revealed the power of combining many predictions in order to obtain an accurate prediction. Ensemble methods manifest this concept in machine learning challenges, where they result in improved predictive performance compared to a single model. In addition, when the computational cost of the participating inducers is low (e.g., decision tree), ensemble models are often very efficient.

Identifying predictors of antimicrobial exposure in hospitalized patients using a machine learning approach

“…This prediction is improved using predictions from earlier nodes of the tree (which occurs recursively up the tree). Although CB regression is consistent with Quinlan’s M5 model tree, it generalizes the model by adding boosting (Bühlmann and Hothorn ; Sagi and Rokach ) and instance‐based corrections. We tuned our CB model empirically to find the best committees C′ and neighbours N parameters using various parameter combinations.…”

Quorums enable optimal pooling of independent judgements

“…Our work may additionally have the potential to inform work in the design of artificial decisionmaking systems machine learning and robotics; for example, in the field of ensemble learning, in which predictions from multiple weak classifiers such as neural networks are combined to improve decision accuracy, variable quorums, referred to as 'threshold shift', are used [e.g. [47]; however majority voting is still among the simplest and most ubiquitous vote fusion rules discussed [ [48,49]]. Hence, we suggest that our simple perspective on how to combine votes may also yield technological insight.…”

Improved deep learning model for differentiating novel coronavirus pneumonia and influenza pneumonia

“…By is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint taking sum of the predicted probabilities for all the lesions of a patient and then normalized between NCP and IP, we got the patient level classification. This simple averaging step could be considered as a model ensemble (22) for patient level classification.…”