Progressive sampling-based Bayesian optimization for efficient and automatic machine learning model selection

Zeng, Xiaoyan; Luo, Gang

doi:10.1007/s13755-017-0023-z

Cited by 71 publications

(57 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Instead of switching between lower fidelities and the highest fidelity, it is possible to perform HPO on a subset of the original data and extract the best-performing configurations in order to use them as an initial design for HPO on the full dataset [152]. To speed up solutions to the CASH problem, it is also possible to iteratively remove entire algorithms (and their hyperparameters) from the configuration space based on poor performance on small dataset subsets [159].…”

Section: Bandit-based Algorithm Selection Methodsmentioning

confidence: 99%

“…1.4), Bayesian optimization with meta-learning (see Chap. 2), and Bayesian optimization taking the pipeline structure into account [159,160]. Furthermore, many recent developments in Bayesian optimization do not directly target HPO, but can often be readily applied to HPO, such as new acquisition functions, new models and kernels, and new parallelization schemes.…”

Section: Bayesian Optimizationmentioning

confidence: 99%

“…Another possibility is to use a separate holdout set to assess configurations found by HPO to avoid bias towards the standard validation set [108,159]. Different approximations of the generalization performance can lead to different test performances [108], and there have been reports that several resampling strategies can result in measurable performance differences for HPO of support vector machines [150].…”

Section: Overfitting and Generalizationmentioning

confidence: 99%

See 2 more Smart Citations

Hyperparameter Optimization

Feurer¹,

Hutter²

2019

The Springer Series on Challenges in Machine Learning

887

496

View full text Add to dashboard Cite

Recent interest in complex and computationally expensive machine learning models with many hyperparameters, such as automated machine learning (AutoML) frameworks and deep neural networks, has resulted in a resurgence of research on hyperparameter optimization (HPO). In this chapter, we give an overview of the most prominent approaches for HPO. We first discuss blackbox function optimization methods based on model-free methods and Bayesian optimization. Since the high computational demand of many modern machine learning applications renders pure blackbox optimization extremely costly, we next focus on modern multi-fidelity methods that use (much) cheaper variants of the blackbox function to approximately assess the quality of hyperparameter settings. Lastly, we point to open problems and future research directions. 1.1 Introduction Every machine learning system has hyperparameters, and the most basic task in automated machine learning (AutoML) is to automatically set these hyperparameters to optimize performance. Especially recent deep neural networks crucially depend on a wide range of hyperparameter choices about the neural network's architecture, regularization, and optimization. Automated hyperparameter optimization (HPO) has several important use cases; it can • reduce the human effort necessary for applying machine learning. This is particularly important in the context of AutoML.

show abstract

Section: Bandit-based Algorithm Selection Methodsmentioning

confidence: 99%

Section: Bayesian Optimizationmentioning

confidence: 99%

Section: Overfitting and Generalizationmentioning

confidence: 99%

See 1 more Smart Citation

Hyperparameter Optimization

Feurer¹,

Hutter²

2019

The Springer Series on Challenges in Machine Learning

887

496

View full text Add to dashboard Cite

show abstract

“…Sequential model-based optimization methods [7,8] and particularly Bayesian optimization methods [9][10][11][12][13][14] were also used in the hyper-parameter optimization. Bayesian search uses a completely different approach compared to the grid search and random search, as the latter ignores the information of the previous search point in the search process, but Bayesian search makes full use of the previous search point information.…”

Section: Introductionmentioning

confidence: 99%

The Tabu_Genetic Algorithm: A Novel Method for Hyper-Parameter Optimization of Learning Algorithms

Guo

et al. 2019

Electronics

View full text Add to dashboard Cite

Machine learning algorithms have been widely used to deal with a variety of practical problems such as computer vision and speech processing. But the performance of machine learning algorithms is primarily affected by their hyper-parameters, as without good hyper-parameter values the performance of these algorithms will be very poor. Unfortunately, for complex machine learning models like deep neural networks, it is very difficult to determine their hyper-parameters. Therefore, it is of great significance to develop an efficient algorithm for hyper-parameter automatic optimization. In this paper, a novel hyper-parameter optimization methodology is presented to combine the advantages of a Genetic Algorithm and Tabu Search to achieve the efficient search for hyper-parameters of learning algorithms. This method is defined as the Tabu_Genetic Algorithm. In order to verify the performance of the proposed algorithm, two sets of contrast experiments are conducted. The Tabu_Genetic Algorithm and other four methods are simultaneously used to search for good values of hyper-parameters of deep convolutional neural networks. Experimental results show that, compared to Random Search and Bayesian optimization methods, the proposed Tabu_Genetic Algorithm finds a better model in less time. Whether in a low-dimensional or high-dimensional space, the Tabu_Genetic Algorithm has better search capabilities as an effective method for finding the hyper-parameters of learning algorithms. The presented method in this paper provides a new solution for solving the hyper-parameters optimization problem of complex machine learning models, which will provide machine learning algorithms with better performance when solving practical problems.

show abstract

“…An example hyper-parameter is the number of hidden layers in a deep neural network. Different combinations of algorithms and hyper-parameter values often impact model accuracy by 40% or more [71] and model building cost by several orders of magnitude [80]. According to the “no free lunch” theorem [76], no single combination performs well on model accuracy for every modeling problem.…”

Section: Introductionmentioning

confidence: 99%

Toward a Progress Indicator for Machine Learning Model Building and Data Mining Algorithm Execution

Luo

2017

SIGKDD Explor. Newsl.

Self Cite

View full text Add to dashboard Cite

For user-friendliness, many software systems offer progress indicators for long-duration tasks. A typical progress indicator continuously estimates the remaining task execution time as well as the portion of the task that has been finished. Building a machine learning model often takes a long time, but no existing machine learning software supplies a non-trivial progress indicator. Similarly, running a data mining algorithm often takes a long time, but no existing data mining software provides a nontrivial progress indicator. In this article, we consider the problem of offering progress indicators for machine learning model building and data mining algorithm execution. We discuss the goals and challenges intrinsic to this problem. Then we describe an initial framework for implementing such progress indicators and two advanced, potential uses of them, with the goal of inspiring future research on this topic.

show abstract

Progressive sampling-based Bayesian optimization for efficient and automatic machine learning model selection

Cited by 71 publications

References 30 publications

Hyperparameter Optimization

Hyperparameter Optimization

The Tabu_Genetic Algorithm: A Novel Method for Hyper-Parameter Optimization of Learning Algorithms

Toward a Progress Indicator for Machine Learning Model Building and Data Mining Algorithm Execution

Contact Info

Product

Resources

About