AdaSwarm: Augmenting Gradient-Based Optimizers in Deep Learning With Swarm Intelligence

Mohapatra, Rohan; Saha, Snehanshu; Coello, Carlos A. Coello; Bhattacharya, Anwesh; Dhavala, Soma S.; Saha, Sriparna

doi:10.1109/tetci.2021.3083428

Cited by 19 publications

(3 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…e ways are proposed to discover operations in the early stages of aerospace design [6]. In [7], the authors described a random multimodal deep learning (RMDL) which is an ensemble system to break the problem of finding a stylish deep learning structure.…”

Section: Related Workmentioning

confidence: 99%

Bio-Optimization of Deep Learning Network Architectures

Shanmugavadivu

Rani

Chitra

et al. 2022

Security and Communication Networks

View full text Add to dashboard Cite

Deep learning is reaching new heights as a result of its cutting-edge performance in a variety of fields, including computer vision, natural language processing, time series analysis, and healthcare. Deep learning is implemented using batch and stochastic gradient descent methods, as well as a few optimizers; however, this led to subpar model performance. However, there is now a lot of effort being done to improve deep learning’s performance using gradient optimization methods. The suggested work analyses convolutional neural networks (CNN) and deep neural networks (DNN) using several cutting-edge optimizers to enhance the performance of architectures. This work uses specific optimizers (SGD, RMSprop, Adam, Adadelta, etc.) to enhance the performance of designs using different types of datasets for result matching. A thorough report on the optimizers’ performance across a variety of architectures and datasets finishes the study effort. This research will be helpful to researchers in developing their framework and appropriate architecture optimizers. The proposed work involves eight new optimizers using four CNN and DNN architectures. The experimental results exploit breakthrough results for improving the efficiency of CNN and DNN architectures using various datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

Bio-Optimization of Deep Learning Network Architectures

Shanmugavadivu

Rani

Chitra

et al. 2022

Security and Communication Networks

View full text Add to dashboard Cite

show abstract

“…The spectrum of problems in which EAs can be used is very wide. EAs have been traditionally applied to optimize neural networks (Iba, 2018), but their usage in DL networks to improve DL networks (Martinez et al, 2021), to train them (Mohapatra et al, 2021), and to create new DL networks from scratch (Elsken et al, 2019b) is more recent. The use of EA's is mainly oriented towards optimizing a complete network.…”

Section: Introductionmentioning

confidence: 99%

EvoPruneDeepTL: An evolutionary pruning model for transfer learning based deep neural networks

et al. 2023

View full text Add to dashboard Cite

Section: Introductionmentioning

confidence: 99%

EvoPruneDeepTL: An Evolutionary Pruning Model for Transfer Learning based Deep Neural Networks

Poyatos¹,

Molina²,

Martinez³

et al. 2022

Preprint

View full text Add to dashboard Cite

In recent years, Deep Learning models have shown a great performance in complex optimization problems. They generally require large training datasets, which is a limitation in most practical cases. Transfer learning allows importing the first layers of a pre-trained architecture and connecting them to fully-connected layers to adapt them to a new problem. Consequently, the configuration of the these layers becomes crucial for the performance of the model. Unfortunately, the optimization of these models is usually a computationally demanding task. One strategy to optimize Deep Learning models is the pruning scheme. Pruning methods are focused on reducing the complexity of the network, assuming an expected performance penalty of the model once pruned. However, the pruning could potentially be used to improve the performance, using an optimization algorithm to identify and eventually remove unnecessary connections among neurons. This work proposes EvoPruneDeepTL, an evolutionary pruning model for Transfer Learning based Deep Neural Networks which replaces the last fully-connected layers with sparse layers optimized by a genetic algorithm. Depending on its solution encoding strategy, our proposed model can either perform optimized pruning or feature selection over the densely connected part of the neural network. We carry out different experiments with several datasets to assess the benefits of our proposal. Results show the contribution of EvoPruneDeepTL and feature selection to the overall computational efficiency of the network as a result of the optimization process. In particular, the accuracy is improved reducing at the same time the number of active neurons in the final layers.

show abstract

AdaSwarm: Augmenting Gradient-Based Optimizers in Deep Learning With Swarm Intelligence

Cited by 19 publications

References 37 publications

Bio-Optimization of Deep Learning Network Architectures

Bio-Optimization of Deep Learning Network Architectures

EvoPruneDeepTL: An evolutionary pruning model for transfer learning based deep neural networks

EvoPruneDeepTL: An Evolutionary Pruning Model for Transfer Learning based Deep Neural Networks

Contact Info

Product

Resources

About