Optimal hyperparameter tuning of convolutional neural networks based on the parameter-setting-free harmony search algorithm

Lee, Wooyoung; Park, Seung Min; Sim, Kwee-Bo

doi:10.1016/j.ijleo.2018.07.044

Cited by 98 publications

(60 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Lee et al 41 proposed a method to tune hyperparameter in the feature extraction step of a convolutional neural network (CNN), the hyperparameter, and harmony memory (HM) are adjusted using a parameter‐setting‐free harmony search (PSF‐HS) algorithm, which is a metaheuristic optimization method. The simulation results showed that by tuning the hyperparameters of a CNN, we can reduce the number of weights and biases that need to be trained and improve classification accuracy.…”

Section: Lf‐dcwqpso Algorithmmentioning

confidence: 99%

An improved quantum particle swarm photovoltaic multi‐peak mPPT method combined with Lévy flight

Chen

Zhang

et al. 2020

Energy Science & Engineering

View full text Add to dashboard Cite

show abstract

Section: Lf‐dcwqpso Algorithmmentioning

confidence: 99%

An improved quantum particle swarm photovoltaic multi‐peak mPPT method combined with Lévy flight

Chen

Zhang

et al. 2020

Energy Science & Engineering

View full text Add to dashboard Cite

show abstract

“…Lee ve arkadaşlarının [58] çalışmalarında veri kümelerine göre farklı boyutlarda KSA seçilmiş ve konvolüsyon katmanında filtre boyutu, filtre sayısı ve aralık değeri; ortaklama katmanında filtre boyutu ve aralık değeri optimize edilmiştir.…”

Section: Harmonik Aramaunclassified

Konvolüsyonel Sinir Ağlarında Hiper-Parametre Optimizasyonu Yöntemlerinin İncelenmesi

Gülcü

Kuş

2019

Gazi Üniversitesi Fen Bilimleri Dergisi Part C: Tasarım Ve Teknoloji

View full text Add to dashboard Cite

In this study, we provide a review on the meta-heuristic methods like Genetic Algorithms, Particle Swarm Optimization, Differential Evolution and Bayes Optimization that have been used extensively to optimize hyper-parameters in Convolutional Neural Networks (CNN). We highlight the hyper-parameters that have been selected to be optimized in those studies along with the value domains of those parameters. These studies reveal that the number of layers, number of kernels and size of those kernels at each layer, learning rate and the batch size are among the hyper-parameters that affect the performance of the CNNs the most. Figure A. structure of convolutional neural networks Purpose: In this study, meta-heuristic methods that have been used to optimize convolutional neural networks are investigated. A performance comparison of these methods on different image datasets has been presented. The advantages and disadvantages of the optimization approaches have been presented with the aim of providing the user important points that should be considered during hyper-parameter selection process. Results: The definiton of "the best" set of hyper-parameters in convolutional neural networks depends on the problem or in this case, on the dataset. But it is clear from the studies that the selection of some parameters directly affect the performance of the networks. Number of layers, number of filters in each layer and size of each filter, regularization method, learning rate and batch size are among the most important parameters. It is easy to conclude that Genetic Algorithms (GA) are the most widely studied techniques used in hyper-parameter optimizaton. This is due to the fact that they yield successful results in most of the studies. While selecting the optimization method, one should consider the size of the problem, available computational budget and time. In addition, accuracy expectations should also be taken into account. For the problems with small hyper-parameter search space, methods like Grid Search would be sufficient, but for the problems with large search space, meta-heuristic methods would be more convenient. Conclusion: In this study, the effect of hyper-parameter optimization methods on classification performance is investigated. GA and Particle Swarm Optimization (PSO) methods are the two most-widely used meta-heuristics for hyper-parameter optimization. The computational burden of these methods can be justified with the accuracy improvement achieved with them. If the computational resources are limited, and it is desired to obtain good results in reasonable amount of time, then other methods like TPE and SMAC would be good choices.

show abstract

“…Differential Evolution (DE) and Harmonic Search (HS) are among the other meta-heuristics that have been used for optimizing CNN hyper-parameters (see [12], [13]). In addition, reinforcement learning (RL) [14], [15] method is also being used for finding the optimal architecture and hyperparameters of CNNs (see [16], [17]).…”

Section: Introductionmentioning

confidence: 99%

Hyper-Parameter Selection in Convolutional Neural Networks Using Microcanonical Optimization Algorithm

Gülcü

Kuş

2020

IEEE Access

View full text Add to dashboard Cite

The success of Convolutional Neural Networks is highly dependent on the selected architecture and the hyper-parameters. The need for the automatic design of the networks is especially important for complex architectures where the parameter space is so large that trying all possible combinations is computationally infeasible. In this study, Microcanonical Optimization algorithm which is a variant of Simulated Annealing method is used for hyper-parameter optimization and architecture selection for Convolutional Neural Networks. To the best of our knowledge, our study provides a first attempt at applying Microcanonical Optimization for this task. The networks generated by the proposed method is compared to the networks generated by Simulated Annealing method in terms of both accuracy and size using six widely-used image recognition datasets. Moreover, a performance comparison using Tree Parzen Estimator which is a Bayesion optimization-based approach is also presented. It is shown that the proposed method is able to achieve competitive classification results with the state-of-the-art architectures. When the size of the networks is also taken into account, one can see that the networks generated by Microcanonical Optimization method contain far less parameters than the state-of-the-art architectures. Therefore, the proposed method can be preferred for automatically tuning the networks especially in situations where fast training is as important as the accuracy. INDEX TERMS Convolutional neural networks, hyper-parameter optimization, microcanonical optimization, tree Parzen estimator.

show abstract

Optimal hyperparameter tuning of convolutional neural networks based on the parameter-setting-free harmony search algorithm

Cited by 98 publications

References 11 publications

An improved quantum particle swarm photovoltaic multi‐peak mPPT method combined with Lévy flight

An improved quantum particle swarm photovoltaic multi‐peak mPPT method combined with Lévy flight

Konvolüsyonel Sinir Ağlarında Hiper-Parametre Optimizasyonu Yöntemlerinin İncelenmesi

Hyper-Parameter Selection in Convolutional Neural Networks Using Microcanonical Optimization Algorithm

Contact Info

Product

Resources

About