Evolution of Activation Functions: An Empirical Investigation

Nader, Andrew; Azar, Danielle

doi:10.1145/3464384

Cited by 9 publications

(4 citation statements)

References 32 publications

(27 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(3) The operation is performed independently for each input channel, resulting in an output with the same number of channels as the input. Here, Y i,j,k is the value of the output feature map at position (i, j) and channel k, X The proposed model incorporates Swish activation functions [28], a choice made due to their well-known smoothness characteristics and effectiveness in enhancing overall model performance. Subsequently, a 1x1 convolutional bottleneck further processes the features.…”

Section: B Adaptivedrnet With Ml-attention: Model Architecturementioning

confidence: 99%

Recognition of Human Interactions in Still Images using AdaptiveDRNet with Multi-level Attention

Dey,

Biswas,

2023

IJACSA

View full text Add to dashboard Cite

Human-Human Interaction Recognition (H2HIR) is a multidisciplinary field that combines computer vision, deep learning, and psychology. Its primary objective is to decode and understand the intricacies of human-human interactions. H2HIR holds significant importance across various domains as it enables machines to perceive, comprehend, and respond to human social behaviors, gestures, and communication patterns. This study aims to identify human-human interactions from just one frame, i.e. from an image. Diverging from the realm of video-based interaction recognition, a well-established research domain that relies on the utilization of spatio-temporal information, the complexity of the task escalates significantly when dealing with still images due to the absence of these intrinsic spatio-temporal features. This research introduces a novel deep learning model called AdaptiveDRNet with Multi-level Attention to recognize Human-Human (H2H) interactions. Our proposed method demonstrates outstanding performance on the Human-Human Interaction Image dataset (H2HID), encompassing 4049 meticulously curated images representing fifteen distinct human interactions and on the publicly accessible HII and HIIv2 related benchmark datasets. Notably, our proposed model excels with a validation accuracy of 97.20% in the classification of human-human interaction images, surpassing the performance of EfficientNet, InceptionResNetV2, NASNet Mobile, ConvXNet, ResNet50, and VGG-16 models. H2H interaction recognition's significance lies in its capacity to enhance communication, improve decision-making, and ultimately contribute to the well-being and efficiency of individuals and society as a whole.

show abstract

Section: B Adaptivedrnet With Ml-attention: Model Architecturementioning

confidence: 99%

Recognition of Human Interactions in Still Images using AdaptiveDRNet with Multi-level Attention

Dey,

Biswas,

2023

IJACSA

View full text Add to dashboard Cite

show abstract

“…A recent work by [17] focused on evolving AFs for neural networks. Their work differs in several aspects from our novel coevolutionary algorithm:…”

Section: Previous Workmentioning

confidence: 99%

Evolution of activation functions for deep learning-based image classification

Lapid

Sipper

2022

Proceedings of the Genetic and Evolutionary Computation Conference Companion

View full text Add to dashboard Cite

Activation functions (AFs) play a pivotal role in the performance of neural networks. The Rectified Linear Unit (ReLU) is currently the most commonly used AF. Several replacements to ReLU have been suggested but improvements have proven inconsistent. Some AFs exhibit better performance for specific tasks, but it is hard to know a priori how to select the appropriate one(s). Studying both standard fully connected neural networks (FCNs) and convolutional neural networks (CNNs), we propose a novel, three-population, coevolutionary algorithm to evolve AFs, and compare it to four other methods, both evolutionary and non-evolutionary. Tested on four datasets-MNIST, FashionMNIST, KMNIST, and USPS-coevolution proves to be a performant algorithm for finding good AFs and AF architectures. CCS CONCEPTS• Computing methodologies → Neural networks; Image processing; • Theory of computation → Evolutionary algorithms.

show abstract

“…e.g. an evolutionary approach was used to evolve the optimal activation function in [35,[100][101][102][103][104][105][106][107][108][109][110][111][112][113][114][115][116] and grid search using artificial data was used in [117]. Another search for the optimal activation functions was presented in [49] where several simple activation functions were found to perform remarkably well.…”

Section: Literature Reviewmentioning

confidence: 99%

“…Another search for the optimal activation functions was presented in [49] where several simple activation functions were found to perform remarkably well. These automatic approaches might be used for evolving the activation functions (e.g., [100,105]) or for selecting the optimal activation function for a given neuron (e.g., [108,118]). While evolved activation function may perform well for a given problem, they also might be very complex -e.g., evolved activation functions in [105].…”

Section: Literature Reviewmentioning

confidence: 99%

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

Kunc

Kléma²

2019

Preprint

View full text Add to dashboard Cite

Motivation: Gene expression profiling was made cheaper by the NIH LINCS program that profiles only ∼1, 000 selected landmark genes and uses them to reconstruct the whole profile. The D-GEX method employs neural networks to infer the whole profile. However, the original D-GEX can be further significantly improved. Results: We have analyzed the D-GEX method and determined that the inference can be improved using a logistic sigmoid activation function instead of the hyperbolic tangent. Moreover, we propose a novel transformative adaptive activation function that improves the gene expression inference even further and which generalizes several existing adaptive activation functions. Our improved neural network achieves average mean absolute error of 0.1340 which is a significant improvement over our reimplementation of the original D-GEX which achieves average mean absolute error 0.1637

show abstract

Evolution of Activation Functions: An Empirical Investigation

Cited by 9 publications

References 32 publications

Recognition of Human Interactions in Still Images using AdaptiveDRNet with Multi-level Attention

Recognition of Human Interactions in Still Images using AdaptiveDRNet with Multi-level Attention

Evolution of activation functions for deep learning-based image classification

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

Contact Info

Product

Resources

About