Multi-Objective Hyperparameter Optimization in Machine Learning—An Overview

Abstract

Hyperparameter optimization constitutes a large part of typical modern machine learning workflows. This arises from the fact that machine learning methods and corresponding preprocessing steps often only yield optimal performance when hyperparameters are properly tuned. But in many applications, we are not only interested in optimizing ML pipelines solely for predictive accuracy; additional metrics or constraints must be considered when determining an optimal configuration, resulting in a multi-objective optimization problem. This is often neglected in practice, due to a lack of knowledge and readily available software implementations for multi-objective hyperparameter optimization. In this work, we introduce the reader to the basics of multi-objective hyperparameter optimization and motivate its usefulness in applied ML. Furthermore, we provide an extensive survey of existing optimization strategies, both from the domain of evolutionary algorithms and Bayesian optimization . We illustrate the utility of MOO in several specific ML applications, considering objectives such as operating conditions, prediction time, sparseness, fairness, interpretability and robustness.

Prediction of bone formation rate of bioceramics using machine learning and image analysis

“…Other feature selection methods select variables to optimise statistics, such as the coefficient of determination, but predictive ability for external data is not considered and such data be overfitted. 18,19 Since the Boruta algorithm does not optimise statistics, the probability of overfitting is low. It uses a feature selection method based on the feature importance in random forests (RF), a regression analysis method.…”

Identification of rail cracks based on path graph features and SVM

“…Each column shows the average rate of finding different crack types within a specific channel over five trials. 19…”

Time Series Trends Forecasting for Manufacturing Enterprises in the Digital Age

“…This model, adept at capturing trends and seasonal variations in time series data, yields dependable forecasts for pivotal business metrics such as production and sales (Kontopoulou et al, 2023). Nonetheless, ARIMA exhibits limitations in handling nonlinear relationships and intricate dynamics, particularly in the dynamic milieu of the digital economy, necessitating further refinement for enhanced applicability (Karl et al, 2023). Secondly, emerging deep learning models leveraging attention mechanisms, such as the Informer model, have gained traction in analyzing manufacturing performance in the digital economy.…”