Evolution of Activation Functions: An Empirical Investigation

Nader, Andrew; Azar, Danielle

doi:10.48550/arxiv.2105.14614

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their results demonstrate that by replacing the standard ReLU activation function with these newly evolved functions improves the performance of neural networks on the CIFAR datasets. A similar approach is also investigated in [50], where the authors apply genetic programming to search for new activation functions. Through their experimental analysis, the authors demonstrate that their new activation functions outperform several standard functions on multivariate classification problems.…”

Section: Related Workmentioning

confidence: 99%

NeuroSCA: Evolving Activation Functions for Side-Channel Analysis

et al. 2023

View full text Add to dashboard Cite

The choice of activation functions can significantly impact the performance of neural networks. Due to an ever-increasing number of new activation functions being proposed in the literature, selecting the appropriate activation function becomes even more difficult. Consequently, many researchers approach this problem from a different angle, in which instead of selecting an existing activation function, an appropriate activation function is evolved for the problem at hand. In this paper, we demonstrate that evolutionary algorithms can evolve new activation functions for side-channel analysis (SCA), outperforming ReLU and other activation functions commonly applied to that problem. More specifically, we use Genetic Programming to define and explore candidate activation functions (neuroevolution) in the form of mathematical expressions that are gradually improved. Experiments with the ASCAD database show that this approach is highly effective compared to results obtained with standard activation functions and that it can match the stateof-the-art results from the literature. More precisely, the obtained results for the ASCAD fixed key dataset demonstrate that the evolved activation functions can improve the current state-of-the-art by achieving a guessing entropy of 287 for the Hamming weight model and 115 for the Identity leakage model, compared to 447 and 120 obtained in the literature.

show abstract

Section: Related Workmentioning

confidence: 99%