Pareto-Based Multiobjective Machine Learning: An Overview and Case Studies

Abstract

Abstract-Machine learning is inherently a multiobjective task. Traditionally, however, either only one of the objectives is adopted as the cost function or multiple objectives are aggregated to a scalar cost function. This can be mainly attributed to the fact that most conventional learning algorithms can only deal with a scalar cost function. Over the last decade, efforts on solving machine learning problems using the Pareto-based multiobjective optimization methodology have gained increasing impetus, particularly due to the great success of multiobjective optimization using evolutionary algorithms and other population-based stochastic search methods. It has been shown that Pareto-based multiobjective learning approaches are more powerful compared to learning algorithms with a scalar cost function in addressing various topics of machine learning, such as clustering, feature selection, improvement of generalization ability, knowledge extraction, and ensemble generation. One common benefit of the different multiobjective learning approaches is that a deeper insight into the learning problem can be gained by analyzing the Pareto front composed of multiple Pareto-optimal solutions. This paper provides an overview of the existing research on multiobjective machine learning, focusing on supervised learning. In addition, a number of case studies are provided to illustrate the major benefits of the Pareto-based approach to machine learning, e.g., how to identify interpretable models and models that can generalize on unseen data from the obtained Pareto-optimal solutions. Three approaches to Pareto-based multiobjective ensemble generation are compared and discussed in detail. Finally, potentially interesting topics in multiobjective machine learning are suggested.

Evolving ensembles in multi-objective genetic programming for classification with unbalanced data

“…In real-world classification tasks, determining a good weighting coefficient can be a lengthy trail and error process, requiring multiple optimisation runs with different weighting coefficients. Evolutionary multi-objective optimisation (EMO) offers a useful solution to the problem of optimising multiple conflicting objectives [17][18] [6]. The aim of EMO is to simultaneously evolve a front of the best trade-off solutions along the objectives in a single optimisation run.…”

“…However, these two learning objectives are usually in conflict; increasing the accuracy of one class can result in lower accuracy on the other. Evolutionary multi-objective optimisation (EMO) is a useful technique to capture this trade-off in the learning process [17][18] [6]. EMO is often advantageous over canonical (single-objective) optimisation techniques because a front of the best tradeoff (non-dominated) solutions along the objectives can be evolved simultaneously in a single optimisation run, without requiring the objective preference to be specified a priori.…”

“…Another advantage of simultaneously evolving a set of classifiers along the minority and majority class trade-off frontier is that the combined classification ability of these non-dominated solutions can be used in a competitive or voting scenario, as these evolved solutions have as much information about a given problem as any single individual on the front [17]. The second goal of this paper is to adapt the MOGP approach to evolve an ensemble of nondominated solutions for classification, using negative correlation learning (NCL) in the fitness to ensure diversity among the ensemble members.…”

“…The second goal of this paper is to adapt the MOGP approach to evolve an ensemble of nondominated solutions for classification, using negative correlation learning (NCL) in the fitness to ensure diversity among the ensemble members. NCL has been shown to successfully evolve accurate and diverse ensembles of neural networks for classification [17][1] [5]. We compare the classification performance of the evolved MOGP ensemble to existing GP approaches and two popular machine learning algorithms, Naive Bayes and Support Vector Machines, on the tasks.…”

A multi-objective genetic optimization of interpretability-oriented fuzzy rule-based classifiers

“…A separate group are the Pareto-based multi-objective optimization methods [47], [48], [49], including the methods based on evolutionary algorithms (e.g. [50], [51]).…”