Neural Network Hyperparameter Tuning based on Improved Genetic Algorithm

Xiang, Wei; Zhining, You

doi:10.1145/3373509.3373554

Cited by 6 publications

(3 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There exists many optimization algorithms. Classic approaches, such as random search (Bergstra and Bengio, 2012 ), Bayesian model (Snoek et al, 2012 ), and evolutional algorithms (Xiang and Zhining, 2019 ), are generally time consuming and may not converge. In recent years, gradient-descent based optimization methods have made it possible to directly optimize hyperparameters in the training loop, such as bilevel optimization (Franceschi et al, 2018 ).…”

Section: Conclusion and Discussionmentioning

confidence: 99%

Efficient training of spiking neural networks with temporally-truncated local backpropagation through time

2023

View full text Add to dashboard Cite

Directly training spiking neural networks (SNNs) has remained challenging due to complex neural dynamics and intrinsic non-differentiability in firing functions. The well-known backpropagation through time (BPTT) algorithm proposed to train SNNs suffers from large memory footprint and prohibits backward and update unlocking, making it impossible to exploit the potential of locally-supervised training methods. This work proposes an efficient and direct training algorithm for SNNs that integrates a locally-supervised training method with a temporally-truncated BPTT algorithm. The proposed algorithm explores both temporal and spatial locality in BPTT and contributes to significant reduction in computational cost including GPU memory utilization, main memory access and arithmetic operations. We thoroughly explore the design space concerning temporal truncation length and local training block size and benchmark their impact on classification accuracy of different networks running different types of tasks. The results reveal that temporal truncation has a negative effect on the accuracy of classifying frame-based datasets, but leads to improvement in accuracy on event-based datasets. In spite of resulting information loss, local training is capable of alleviating overfitting. The combined effect of temporal truncation and local training can lead to the slowdown of accuracy drop and even improvement in accuracy. In addition, training deep SNNs' models such as AlexNet classifying CIFAR10-DVS dataset leads to 7.26% increase in accuracy, 89.94% reduction in GPU memory, 10.79% reduction in memory access, and 99.64% reduction in MAC operations compared to the standard end-to-end BPTT. Thus, the proposed method has shown high potential to enable fast and energy-efficient on-chip training for real-time learning at the edge.

show abstract

Section: Conclusion and Discussionmentioning

confidence: 99%

Efficient training of spiking neural networks with temporally-truncated local backpropagation through time

2023

View full text Add to dashboard Cite

show abstract

“…An optimized CNN is designed to do the image recognition work, which accuracy is improved from 90% to more than 98%. Wei and You (2019) used the improved genetic algorithm to optimize a series of super parameters in the fully connected neural network in CNN. The accuracy of their model can reach 98.81%, which is higher than 98.4% of the official sample model.…”

Section: Introductionmentioning

confidence: 99%

Application of LSTM model optimized by individual-ordering-based adaptive genetic algorithm in stock forecasting

Zeng

et al. 2022

IJICC

View full text Add to dashboard Cite

PurposeTo improve the accuracy of stock price trend prediction in the field of quantitative financial trading, this paper takes the prediction accuracy as the goal and avoid the enormous number of network structures and hyperparameter adjustments of long-short-term memory (LSTM).Design/methodology/approachIn this paper, an adaptive genetic algorithm based on individual ordering is used to optimize the network structure and hyperparameters of the LSTM neural network automatically.FindingsThe simulation results show that the accuracy of the rise and fall of the stock outperform than the model with LSTM only as well as other machine learning models. Furthermore, the efficiency of parameter adjustment is greatly higher than other hyperparameter optimization methods.Originality/value(1) The AGA-LSTM algorithm is used to input various hyperparameter combinations into genetic algorithm to find the best hyperparameter combination. Compared with other models, it has higher accuracy in predicting the up and down trend of stock prices in the next day. (2) Adopting real coding, elitist preservation and self-adaptive adjustment of crossover and mutation probability based on individual ordering in the part of genetic algorithm, the algorithm is computationally efficient and the results are more likely to converge to the global optimum.

show abstract

“…They managed to increase the classification accuracy of the CNN from 90% to >98%. Wei and You (2019) adopted GA to improve the fitness of the modified national institute of standards and technology (MNIST) image recognition problem and managed to increase the prediction accuracy from 98.10% to 98.84%. In addition, the authors developed a two-point fusion intermediate crossover and adaptive mutation to optimize the hyperparameters of the fully connected NN.…”

Section: Introductionmentioning

confidence: 99%

Optimizing the neural network hyperparameters utilizing genetic algorithm

Nikbakht

Anitescu

Rabczuk

2021

J. Zhejiang Univ. Sci. A

View full text Add to dashboard Cite

Neural networks (NNs), as one of the most robust and efficient machine learning methods, have been commonly used in solving several problems. However, choosing proper hyperparameters (e.g. the numbers of layers and neurons in each layer) has a significant influence on the accuracy of these methods. Therefore, a considerable number of studies have been carried out to optimize the NN hyperparameters. In this study, the genetic algorithm is applied to NN to find the optimal hyperparameters. Thus, the deep energy method, which contains a deep neural network, is applied first on a Timoshenko beam and a plate with a hole. Subsequently, the numbers of hidden layers, integration points, and neurons in each layer are optimized to reach the highest accuracy to predict the stress distribution through these structures. Thus, applying the proper optimization method on NN leads to significant increase in the NN prediction accuracy after conducting the optimization in various examples.

show abstract

Neural Network Hyperparameter Tuning based on Improved Genetic Algorithm

Cited by 6 publications

References 5 publications

Efficient training of spiking neural networks with temporally-truncated local backpropagation through time

Efficient training of spiking neural networks with temporally-truncated local backpropagation through time

Application of LSTM model optimized by individual-ordering-based adaptive genetic algorithm in stock forecasting

Optimizing the neural network hyperparameters utilizing genetic algorithm

Contact Info

Product

Resources

About