A Review of Random Search Methods

Andradóttir, Sigrún

doi:10.1007/978-1-4939-1384-8_10

Cited by 42 publications

(18 citation statements)

References 83 publications

(97 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nonetheless, Bergstra and Bengio (2012) showed that it is (at least) as good as more advanced versions of random search. More details on these algorithms can be found in Solis and Wets (1981) and Andradóttir (2015).…”

Section: Hyperparameters Selectionmentioning

confidence: 99%

Selecting time-series hyperparameters with the artificial jackknife

Pellegrino¹

2020

Preprint

View full text Add to dashboard Cite

This article proposes a generalisation of the delete-d jackknife to solve hyperparameter selection problems for time series. This novel technique is compatible with dependent data since it substitutes the jackknife removal step with a fictitious deletion, wherein observed datapoints are replaced with artificial missing values. In order to emphasise this point, I called this methodology artificial delete-d jackknife. As an illustration, it is used to regulate vector autoregressions with an elastic-net penalty on the coefficients.A software implementation, ElasticNetVAR.jl, is available on GitHub.

show abstract

Section: Hyperparameters Selectionmentioning

confidence: 99%

Selecting time-series hyperparameters with the artificial jackknife

Pellegrino¹

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…-Random search randomly picks combinations from all possible ones. It may not find a decent combination but is widely adopted in industry for the high-efficiency [1]. -Bayesian optimization uses a Gaussian process to minimize the loss function in order to maximize performance [10].…”

Section: Overall Comparisonmentioning

confidence: 99%

“…Random Search, which is developed based on grid research, set up a grid of arXiv:1907.13359v1 [cs.LG] 31 Jul 2019 hyper-parameter values and selects random combinations to train the algorithm [2]. Random search method oversteps some disadvantages of grid search such as time-consuming but meanwhile brings a major disadvantage which cannot converge to the global optimum [1]. The randomly selected hyper-parameter combinations cannot guarantee a steady and competitive result.…”

Section: Introductionmentioning

confidence: 99%

Deep Neural Network Hyperparameter Optimization with Orthogonal Array Tuning

Zhang

Chen

Yao

et al. 2019

Communications in Computer and Information Science

View full text Add to dashboard Cite

Deep learning algorithms have achieved excellent performance lately in a wide range of fields (e.g., computer version). However, a severe challenge faced by deep learning is the high dependency on hyperparameters. The algorithm results may fluctuate dramatically under the different configuration of hyper-parameters. Addressing the above issue, this paper presents an efficient Orthogonal Array Tuning Method (OATM) for deep learning hyper-parameter tuning. We describe the OATM approach in five detailed steps and elaborate on it using two widely used deep neural network structures (Recurrent Neural Networks and Convolutional Neural Networks). The proposed method is compared to the state-of-the-art hyper-parameter tuning methods including manually (e.g., grid search and random search) and automatically (e.g., Bayesian Optimization) ones. The experiment results state that OATM can significantly save the tuning time compared to the state-of-the-art methods while preserving the satisfying performance.

show abstract

“…An incomplete list of recent examples includes model reference adaptive search (Hu et al 2007), adaptive search with resampling (Andradóttir and Prudius 2010), gradient-based adaptive stochastic search (Zhou and Bhatnagar 2018), and single observation search (Kiatsupaibul et al 2018). We refer to Andradóttir (2015) and Zabinsky (2015) for reviews of random search methods.…”

Section: Related Workmentioning

confidence: 99%

High-Dimensional Simulation Optimization via Brownian Fields and Sparse Grids

Ding

Tuo

Zhang

2021

Preprint

View full text Add to dashboard Cite

High-dimensional simulation optimization is notoriously challenging. We propose a new sampling algorithm that converges to a global optimal solution and suffers minimally from the curse of dimensionality. The algorithm consists of two stages. First, we take samples following a sparse grid experimental design and approximate the response surface via kernel ridge regression with a Brownian field kernel. Second, we follow the expected improvement strategy-with critical modifications that boost the algorithm's sample efficiencyto iteratively sample from the next level of the sparse grid. Under mild conditions on the smoothness of the response surface and the simulation noise, we establish upper bounds on the convergence rate for both noise-free and noisy simulation samples. These upper bounds deteriorate only slightly in the dimension of the feasible set, and they can be improved if the objective function is known to be of a higher-order smoothness.Extensive numerical experiments demonstrate that the proposed algorithm dramatically outperforms typical alternatives in practice.

show abstract

A Review of Random Search Methods

Cited by 42 publications

References 83 publications

Selecting time-series hyperparameters with the artificial jackknife

Selecting time-series hyperparameters with the artificial jackknife

Deep Neural Network Hyperparameter Optimization with Orthogonal Array Tuning

High-Dimensional Simulation Optimization via Brownian Fields and Sparse Grids

Contact Info

Product

Resources

About