An ensemble of differential evolution and Adam for training feed-forward neural networks

Xue, Yu; Tong, Yiling; Neri, Ferrante

doi:10.1016/j.ins.2022.06.036

Cited by 102 publications

(33 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The aim is to reach the minimum of that cost function by taking small steps in the direction of the negative gradient. One drawback is their tendency to converge easily towards local optima [5]. SGD [6] as one of the most favored gradientbased algorithms for training NNs, also suffers from early convergence.…”

Section: A Convolutional Neural Networkmentioning

confidence: 99%

“…To avert premature convergence, a wide range of adaptive gradient algorithms have been developed that adjust the learning rate in efficient ways, one worth mentioning is Adam. The issue of premature convergence has been tackled in past literature through trying to improve the global search of gradient descent methods [5], [7]. All of the proposed solutions in these respective papers focus on hybridizing the very effective convergence speed of gradient descent with the gradient-less global search of meta-heuristic optimization algorithms.…”

Section: A Convolutional Neural Networkmentioning

confidence: 99%

“…This approach generates the optimal networks for the given problem, however, this research aims to design an algorithm which trains the weights of the network. There were multiple instances in the literature [5], [7], [11] of different researchers combining gradient-based methods with metaheuristic algorithms. This was done to train neural networks by exchanging information between the two types.…”

Section: Population Meta-heuristics For Neural Networkmentioning

confidence: 99%

See 2 more Smart Citations

Genetically Modified Wolf Optimization with Stochastic Gradient Descent for Optimising Deep Neural Networks

Bradicic¹,

Sitarz²,

Olesen³

2023

Preprint

View full text Add to dashboard Cite

When training Convolutional Neural Networks (CNNs) there is a large emphasis on creating efficient optimization algorithms and highly accurate networks. The stateof-the-art method of optimizing the networks is done by using gradient descent algorithms, such as Stochastic Gradient Descent (SGD). However, there are some limitations presented when using gradient descent methods. The major drawback is the lack of exploration, and over-reliance on exploitation. Hence, this research aims to analyze an alternative approach to optimizing neural network (NN) weights, with the use of population-based metaheuristic algorithms. A hybrid between Grey Wolf Optimizer (GWO) and Genetic Algorithms (GA) is explored, in conjunction with SGD; producing a Genetically Modified Wolf optimization algorithm boosted with SGD (GMW-SGD). This algorithm allows for a combination between exploitation and exploration, whilst also tackling the issue of high-dimensionality, affecting the performance of standard metaheuristic algorithms. The proposed algorithm was trained and tested on CIFAR-10 where it performs comparably to the SGD algorithm, reaching high test accuracy, and significantly outperforms standard metaheuristic algorithms.

show abstract

Section: A Convolutional Neural Networkmentioning

confidence: 99%

Section: A Convolutional Neural Networkmentioning

confidence: 99%

Section: Population Meta-heuristics For Neural Networkmentioning

confidence: 99%

See 1 more Smart Citation

Genetically Modified Wolf Optimization with Stochastic Gradient Descent for Optimising Deep Neural Networks

Bradicic¹,

Sitarz²,

Olesen³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Various MHs have been proposed and evaluated over the last decades to solve a colorful palette of challenging problems [10]. However, some of the newly developed metaheuristics do not differ substantially from the general structures of well-reputed and conventional MHs such as Simulated Annealing (SA) [11], Differential Evolution (DE) [12], Genetic Algorithms (GA) [13], Grey-Wolf Optimizer (GWO) [14], and Particle Swarm Optimization (PSO) [15], to mention a few.…”

Section: Introductionmentioning

confidence: 99%

Automatic Design of Metaheuristics for Practical Engineering Applications

et al. 2023

View full text Add to dashboard Cite

It is common to find multiple metaheuristics to solve continuous optimization problems. However, choosing what optimizer may obtain the best results for a given task requires exhaustive evaluations that are highly application-dependent. Besides, it is necessary to find sufficiently good tuning parameters to achieve satisfactory performance with the selected approach. In this context, the automatic design of algorithms, particularly those based on heuristics, has been increasing in popularity in the previous years due to its undoubted relevance nowadays. This paper explores a novel approach based on hyper-heuristics to carefully select population-based search operators and their tuning parameters to generate metaheuristics capable of dealing with a given practical engineering problem. The proposed strategy is assessed using three highly relevant and illustrative problems: training Artificial Neural Networks, designing PID controllers, and modeling a calorimetric phenomenon based on fractional calculus. In addition, we implement three well-known optimization metaheuristics to compare achieved solutions via the proposed hyper-heuristic strategy, namely Particle Swarm Optimization, Genetic Algorithm, and Cuckoo Search. Results from extensive numerical tests prove that the customized metaheuristics are generally superior to the three wellknown algorithms, taking only a few iterations to converge to an optimal solution. This is an excellent indicator of alleviating the effort and expertise required to choose the proper methodology when dealing with real-valued optimization problems.

show abstract

“…A solution was proposed in terms of an ensemble of differential evolution and Adam (EDEAdam), combining both Adam optimizer and differential evolution algorithm, which forms a robust and efficient search mechanism to achieve better results in both global and local search. The integration of these two methods not only helped improve results but also showed faster convergence speed (Xue et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

Grid Arama Yoluyla Monotonik Olmayan Hiperparametre Planlama Sisteminin Yardımcı Öğrenimi

Hameed

2022

jista

View full text Add to dashboard Cite

Gelişmiş sinir ağlarındaki son gelişmeler, yeni uyarlanabilir öğrenme stratejilerine yol açmıştır. Geleneksel öğrenme stratejileri, yavaş yakınsama ve sağlamlık eksikliği gibi birçok sorundan muzdariptir. Potansiyelinden tam olarak yararlanmak için bu sorunların çözülmesi gerekir. Her iki konu da adım boyutu ve genellikle sabit olan ve her ağ katmanıyla ilişkili tüm ağırlıklar için tek tip kalan momentum terimi ile ilgilidir. Bu çalışmada, bu sorunların üstesinden gelmek ve sınıflandırma etkinliğini artırmak için yakın zamanda yayınlanan Değişken Uyarlanabilir Momentumlu Geri Yayılım Algoritması (BPVAM) algoritması önerilmiştir. Çalışma grid arama yaklaşımına dayalı olarak çeşitli hiperparametreler üzerinde yürütülmüş, daha sonra hiperparametrelerin optimal değerleri bu algoritmaları eğitmiştir. Hiperparametrenin eğitim modelleri üzerindeki etkisini değerlendirmek için hiperparametrenin değişen değerlerine sahip altı durum ele alındı. Modelin yakınsama davranışının, doğruluk için ortalama ve standart sapma ve karesel hatanın toplamı (SSE) açısından iyileştirildiği deneysel olarak kanıtlanmıştır. Kapsamlı bir deney seti, BPVAM'nin sağlam ve yüksek verimli bir algoritma olduğunu gösterdi.

show abstract

An ensemble of differential evolution and Adam for training feed-forward neural networks

Cited by 102 publications

References 43 publications

Genetically Modified Wolf Optimization with Stochastic Gradient Descent for Optimising Deep Neural Networks

Genetically Modified Wolf Optimization with Stochastic Gradient Descent for Optimising Deep Neural Networks

Automatic Design of Metaheuristics for Practical Engineering Applications

Grid Arama Yoluyla Monotonik Olmayan Hiperparametre Planlama Sisteminin Yardımcı Öğrenimi

Contact Info

Product

Resources

About