Stochastic Selection of Activation Layers for Convolutional Neural Networks

Abstract

In recent years, the field of deep learning has achieved considerable success in pattern recognition, image segmentation, and many other classification fields. There are many studies and practical applications of deep learning on images, video, or text classification. Activation functions play a crucial role in discriminative capabilities of the deep neural networks and the design of new “static” or “dynamic” activation functions is an active area of research. The main difference between “static” and “dynamic” functions is that the first class of activations considers all the neurons and layers as identical, while the second class learns parameters of the activation function independently for each layer or even each neuron. Although the “dynamic” activation functions perform better in some applications, the increased number of trainable parameters requires more computational time and can lead to overfitting. In this work, we propose a mixture of “static” and “dynamic” activation functions, which are stochastically selected at each layer. Our idea for model design is based on a method for changing some layers along the lines of different functional blocks of the best performing CNN models, with the aim of designing new models to be used as stand-alone networks or as a component of an ensemble. We propose to replace each activation layer of a CNN (usually a ReLU layer) by a different activation function stochastically drawn from a set of activation functions: in this way, the resulting CNN has a different set of activation function layers.

Artificial intelligence for classifying uncertain images by humans in determining choroidal vascular running pattern and comparisons with automated classification between artificial intelligence

“…The Random Forest model had the highest discrimination performance, but all models had sufficient capabilities with high AUC values. It has been reported that the accuracy of CNN models can be improved by combining them [ 30 – 32 ]. Although only one CNN model was used in this study, it may be possible to improve the accuracy further by combining multiple CNN models.…”

Postprocessing for Skin Detection

“…As in [ 29 ], the deep learner for image segmentation is Deeplabv3+ [ 30 ]. Its predecessor, Deeplabv3 [ 31 ], uses atrous convolution [ 32 ], or dilated convolution, to repurpose pretrained networks in order to control the resolution of feature responses without adding extra parameters.…”

“…The encoder–decoder structure of DeepLabv3+ allows it to be built on top of a pretrained CNN, such ResNet50, the architecture used here (internal preliminary investigations showed that both ResNet101 and ResNet34 produced similar results to ResNet50). Following the methodology outlined in reference [ 29 ], all models for skin segmentation were trained on a dataset containing 2000 images using class weighting; the training parameters used in this work, however, are the following: batch size (30), learning rate (0.001), and max epoch (50), with data augmentation set to 30 epochs.…”

“…For comparison purposes, an ensemble of classifiers is designed to fuse three variants of DeepLabv3+ with the variants constructed by substituting all the activation layers with a different activation function, as in reference [ 29 ], so that the diversity of the networks can improve the ensemble performance. According to the experiments in [ 29 ], which investigated many different activation functions using this approach, one of the best-performing ensembles for skin detection is FusAct3, a lightweight ensemble obtained by the fusion of DeepLabv3+ on three variants of the standard ReLU function: MeLU, wMeLU, and PReLU [ 33 ]. Since DeepLabv3+ and the ensemble FusAct3 gain a state-of-the-art performance for this problem, we used these approaches as a baseline of this research.…”

Nonlinear Neural Network Based Forecasting Model for Predicting COVID-19 Cases

“…Maguolo et al [ 49 ] have proposed a weighted resampling based transfer learning algorithm that combines efficient data with the data labelled in its target domain, and then, this scheme integrates the learned classifiers with the integrated data. A novel model has been proposed by Nanni et al [ 50 ], which combines static and dynamic activation functions. This scheme replaces all activation layers of a Convolutional Neural Network (CNN).…”