Hyperparameter optimization for deep neural network models: a comprehensive study on methods and techniques

Abstract

Advancements in computing and storage technologies have significantly contributed to the adoption of deep learning (DL)-based models among machine learning (ML) experts. Although a generic model can be used in the search for a near-optimal solution in any problem domain, what makes these DL models context-sensitive is the combination of the training data and the hyperparameters. Due to the lack of inherent explainability of DL models the Hyperparameter Optimization (HPO) or tuning specific to each model is a combination of art, science, and experience. In this article, we have explored various existing methods or ways to identify the optimal set of values for the hyperparameters specific to the DL models along with the techniques to realize those methods in real-life situations. The article also includes a detailed comparative study among various state-of-the-art HPO techniques using the Keras Tuner tuning toolkit and highlights the observations describing how the model performance can be improved by applying various HPO techniques.

Two-Stage Neural Network Optimization for Robust Solar Photovoltaic Forecasting

“…Traditional hyperparameter tuning techniques have been extended to include gradientbased optimization, evolutionary algorithms, simulated annealing, particle swarm optimization, hyperband, gradient-based optimization, random search, and Bayesian optimization to improve DL models [50,51]. While these methods expand the range of tools available to improve model performance, they often share the same drawbacks: high computational requirements, significant time commitment, and a complexity that can be intimidating to practitioners not trained in ML, such as those working in the energy industry.…”

Load Day-Ahead Automatic Generation Control Reserve Capacity Demand Prediction Based on the Attention-BiLSTM Network Model Optimized by Improved Whale Algorithm

“…Table 5 presents a comparison of the prediction model proposed in this paper with four other models, namely back-propagation neural network (BP), long short-term memory neural network (LSTM), bidirectional long short-term memory neural network (BiLSTM) and Attention-BiLSTM. In addition, since particle swarm algorithms (PSOs) and genetic algorithms (GAs) can also be used for hyperparameter optimization [61], the prediction model in this paper is also compared with PSO-Attention-BiLSTM model and GA-Attention-BiLSTM, and the range of hyperparameter settings is the same for all three optimization models. Table 5 showcases the prediction averages for the six-month period and the overall prediction results for the second half of the year.…”