A survey on multi-objective hyperparameter optimization algorithms for Machine Learning

Abstract

Hyperparameter optimization (HPO) is a necessary step to ensure the best possible performance of Machine Learning (ML) algorithms. Several methods have been developed to perform HPO; most of these are focused on optimizing one performance measure (usually an error-based measure), and the literature on such single-objective HPO problems is vast. Recently, though, algorithms have appeared which focus on optimizing multiple conflicting objectives simultaneously. This article presents a systematic survey of the literature published between 2014 and 2020 on multi-objective HPO algorithms, distinguishing between metaheuristicbased algorithms, metamodel-based algorithms and approaches using a mixture of both. We also discuss the quality metrics used to compare multi-objective HPO procedures, and present future research directions.

Deep Learning and Bayesian Hyperparameter Optimization: A Data-Driven Approach for Diamond Grit Segmentation toward Grinding Wheel Characterization

“…For example, in the simple version of our model, there are more than 8600 hyperparameter combinations. Deep learning hyperparameter optimization (HPO) is an active research area [44][45][46][47][48][49]. In such problems, the goal is to maximize/minimize an expensive objective function whose closed form and its gradients are unknown.…”

General Cyclical Training of Neural Networks

“…Deep neural networks lie at the heart of many of the artificial intelligence applications that are ubiquitous in our society. Over the past several years, methods for training these networks have become more automatic [1,2,3,4,5] but still remain more an art than a science. This paper introduces the high-level concept of general cyclical training as another step in making it easier to optimally train neural networks.…”