Evolution of activation functions for deep learning-based image classification

Lapid, Raz; Sipper, Moshe

doi:10.1145/3520304.3533949

Cited by 12 publications

(5 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Evolution may also be a solution for rendering models more robust. In [57] it was shown that combining different activation functions could be used to increase model accuracy; this approach might also be used for obtaining robustness.…”

Section: Discussion and Concluding Remarksmentioning

confidence: 99%

An Evolutionary, Gradient-Free, Query-Efficient, Black-Box Algorithm for Generating Adversarial Instances in Deep Convolutional Neural Networks

2022

Self Cite

View full text Add to dashboard Cite

Deep neural networks (DNNs) are sensitive to adversarial data in a variety of scenarios, including the black-box scenario, where the attacker is only allowed to query the trained model and receive an output. Existing black-box methods for creating adversarial instances are costly, often using gradient estimation or training a replacement network. This paper introduces Qu ery-Efficient Evolutionary Attack—QuEry Attack—an untargeted, score-based, black-box attack. QuEry Attack is based on a novel objective function that can be used in gradient-free optimization problems. The attack only requires access to the output logits of the classifier and is thus not affected by gradient masking. No additional information is needed, rendering our method more suitable to real-life situations. We test its performance with three different, commonly used, pretrained image-classifications models—Inception-v3, ResNet-50, and VGG-16-BN—against three benchmark datasets: MNIST, CIFAR10 and ImageNet. Furthermore, we evaluate QuEry Attack’s performance on non-differential transformation defenses and robust models. Our results demonstrate the superior performance of QuEry Attack, both in terms of accuracy score and query efficiency.

show abstract

Section: Discussion and Concluding Remarksmentioning

confidence: 99%

An Evolutionary, Gradient-Free, Query-Efficient, Black-Box Algorithm for Generating Adversarial Instances in Deep Convolutional Neural Networks

2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…Although the proposed method can construct activation functions that outperform traditional activation functions, like ReLU, due to the way in which solutions are represented, the method can only construct activation functions of a given structure. A coevolutionary algorithm to evolve new activation functions for standard fully-connected and convolutional neural networks is proposed in [52]. The authors use Cartesian GP to evolve new activation functions, which can evolve new activation functions of various structures.…”

Section: Related Workmentioning

confidence: 99%

NeuroSCA: Evolving Activation Functions for Side-Channel Analysis

et al. 2023

View full text Add to dashboard Cite

The choice of activation functions can significantly impact the performance of neural networks. Due to an ever-increasing number of new activation functions being proposed in the literature, selecting the appropriate activation function becomes even more difficult. Consequently, many researchers approach this problem from a different angle, in which instead of selecting an existing activation function, an appropriate activation function is evolved for the problem at hand. In this paper, we demonstrate that evolutionary algorithms can evolve new activation functions for side-channel analysis (SCA), outperforming ReLU and other activation functions commonly applied to that problem. More specifically, we use Genetic Programming to define and explore candidate activation functions (neuroevolution) in the form of mathematical expressions that are gradually improved. Experiments with the ASCAD database show that this approach is highly effective compared to results obtained with standard activation functions and that it can match the stateof-the-art results from the literature. More precisely, the obtained results for the ASCAD fixed key dataset demonstrate that the evolved activation functions can improve the current state-of-the-art by achieving a guessing entropy of 287 for the Hamming weight model and 115 for the Identity leakage model, compared to 447 and 120 obtained in the literature.

show abstract

“…To fix this problem, some modifications were made which resulted to some of the variants of the ReLU activation function. These activation functions include: Leaky ReLU [21], SELU [22], PReLU [23], GELU [24], ELU [25], RReLU [26], CELU [27] and RELU6 [28].…”

Section: Activation Functionsmentioning

confidence: 99%

The Effect of Activation Functions in a Chest X-ray Image Classification

Nworu

Ekpenyong

John

et al. 2022

Preprint

View full text Add to dashboard Cite

The choice of activation functons is very important in deep learning. This is because activation functions are capable of capturing non-linear patterns in a data. The most popular actiavtion function is the Rectified Linear Unit (ReLU) but it suffers from gradient vanishing problem. Therefore, we examined the modifications of the ReLU activation function to determine its effectiveness (accuracy) and efficiency (time complexity). The effectiveness and efficiency verified by conducting an empirical experiment using X-ray images that contains pneumonia and normal samples. Our experiments show that the modified ReLU, ReLU6 performed better in terms of low generalization error (97.05% training accuracy and 78.21% test accuracy). The sensitivity measure also suggest that the ELU is capable of correctly predicting more than half of the positive cases with 52.14% probability. For efficiency, the GELU shows the lowest training time when compared with other activation functions. This will allow practitioners in this field to choose activation functions based on effectiveness and efficiency JEL Classification: C63 , C67 , I10

show abstract

Evolution of activation functions for deep learning-based image classification

Cited by 12 publications

References 25 publications

An Evolutionary, Gradient-Free, Query-Efficient, Black-Box Algorithm for Generating Adversarial Instances in Deep Convolutional Neural Networks

An Evolutionary, Gradient-Free, Query-Efficient, Black-Box Algorithm for Generating Adversarial Instances in Deep Convolutional Neural Networks

NeuroSCA: Evolving Activation Functions for Side-Channel Analysis

The Effect of Activation Functions in a Chest X-ray Image Classification

Contact Info

Product

Resources

About