An effective algorithm for hyperparameter optimization of neural networks

Diaz, Gonzalo I.; Fokoue-Nkoutche, Achille; Nannicini, Giacomo; Samulowitz, Horst

doi:10.1147/jrd.2017.2709578

Cited by 141 publications

(72 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The performance of SA methods is highly sensitive to the chosen sequence of step sizes {α k } [Hutchison and Spall, 2013]. This mirrors the situation in gradient-based SA methods where the tuning of algorithmic parameters is an active area of research [Diaz et al, 2017, Ilievski et al, 2017, Balaprakash et al, 2018.…”

Section: Stochastic and Sample-average Approximationmentioning

confidence: 96%

Derivative-free optimization methods

2019

View full text Add to dashboard Cite

Dedicated to the memory of Andrew R. Conn for his inspiring enthusiasm and his many contributions to the renaissance of derivative-free optimization methods. AbstractIn many optimization problems arising from scientific, engineering and artificial intelligence applications, objective and constraint functions are available only as the output of a black-box or simulation oracle that does not provide derivative information. Such settings necessitate the use of methods for derivative-free, or zeroth-order, optimization. We provide a review and perspectives on developments in these methods, with an emphasis on highlighting recent developments and on unifying treatment of such problems in the non-linear optimization and machine learning literature. We categorize methods based on assumed properties of the black-box functions, as well as features of the methods. We first overview the primary setting of deterministic methods applied to unconstrained, non-convex optimization problems where the objective function is defined by a deterministic black-box oracle. We then discuss developments in randomized methods, methods that assume some additional structure about the objective (including convexity, separability and general non-smooth compositions), methods for problems where the output of the black-box oracle is stochastic, and methods for handling different types of constraints.

show abstract

Section: Stochastic and Sample-average Approximationmentioning

confidence: 96%

Derivative-free optimization methods

2019

View full text Add to dashboard Cite

show abstract

“…The capabilities of a neural network to make good predictions depends on its architecture and its parameters, it is an essential task to define a well structured network before implementing the model. Parameters which define the model architecture are known as hyperparameters and the process of assessing the best configuration for those parameters is called hyperparameter tuning (Diaz et al, 2017).…”

Section: Use Of Deep Learning Models For Classification Of Olive Oilmentioning

confidence: 99%

Deep Learning Techniques to Improve the Performance of Olive Oil Classification

et al. 2020

View full text Add to dashboard Cite

The olive oil assessment involves the use of a standardized sensory analysis according to the "panel test" method. However, there is an important interest to design novel strategies based on the use of Gas Chromatography (GC) coupled to mass spectrometry (MS), or ion mobility spectrometry (IMS) together with a chemometric data treatment for olive oil classification. It is an essential task in an attempt to get the most robust model over time and, both to avoid fraud in the price and to know whether it is suitable for consumption or not. The aim of this paper is to combine chemical techniques and Deep Learning approaches to automatically classify olive oil samples from two different harvests in their three corresponding classes: extra virgin olive oil (EVOO), virgin olive oil (VOO), and lampante olive oil (LOO). Our Deep Learning model is built with 701 samples, which were obtained from two olive oil campaigns (2014-2015 and 2015-2016). The data from the two harvests are built from the selection of specific olive oil markers from the whole spectral fingerprint obtained with GC-IMS method. In order to obtain the best results we have configured the parameters of our model according to the nature of the data. The results obtained show that a deep learning approach applied to data obtained from chemical instrumental techniques is a good method when classifying oil samples in their corresponding categories, with higher success rates than those obtained in previous works.

show abstract

“…• Stochastic Approximation [21], hill climbing where hyperparameters are individually and sequentially changed • Evolutionary algorithms [20], which randomly start, select the best initial results (parents), and then generate multiple possible outcomes (children), and then repeat the process • Bayesian optimization (BO) [5] which treats the objective function as a random function and uses randomly determined hyperparameters to construct a distribution around the results • Other approaches which do not fit cleanly into these three groups, e.g. Radial Basis Functions [22], Hyberband [23], Nedler-Mead [24], and spectral approaches [25]. Beyond this work, further approaches include extensions of BO and combinations of methods.…”

Section: Ai Hyperparameter Determinationmentioning

confidence: 99%

Easy and Efficient Hyperparameter Optimization to Address Some Artificial Intelligence “ilities”

Bihl¹,

Schoenbeck²,

Steeneck³

et al. 2020

Proceedings of the Annual Hawaii International Conference on System Sciences

View full text Add to dashboard Cite

Artificial Intelligence (AI), has many benefits, including the ability to find complex patterns, automation, and meaning making. Through these benefits, AI has revolutionized image processing among numerous other disciplines. AI further has the potential to revolutionize other domains; however, this will not happen until we can address the "ilities": repeatability, explain-ability, reliability, use-ability, trust-ability, etc. Notably, many problems with the "ilities" are due to the artistic nature of AI algorithm development, especially hyperparameter determination. AI algorithms are often crafted products with the hyperparameters learned experientially. As such, when applying the same algorithm to new problems, the algorithm may not perform due to inappropriate settings. This research aims to provide a straightforward and reliable approach to automatically determining suitable hyperparameter settings when given an AI algorithm. Results, show reasonable performance is possible and end-to-end examples are given for three deep learning algorithms and three different data problems.

show abstract

An effective algorithm for hyperparameter optimization of neural networks

Cited by 141 publications

References 14 publications

Derivative-free optimization methods

Derivative-free optimization methods

Deep Learning Techniques to Improve the Performance of Olive Oil Classification

Easy and Efficient Hyperparameter Optimization to Address Some Artificial Intelligence “ilities”

Contact Info

Product

Resources

About