Linearized sigmoidal activation: A novel activation function with tractable non-linear characteristics to boost representation capability

Bawa, Vivek Singh; Kumar, Vinay

doi:10.1016/j.eswa.2018.11.042

Cited by 45 publications

(16 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figure 5A shows the importance of each hyperparameter obtained from random forest by regressing the model predictive accuracies (obtained by an inner crossvalidation) on all hyperparameter values combinations. Interestingly, the number of filters was overall the most relevant factor, whereas other factors such as learning rate, whose importance has been claimed in the literature as critically important (Maas et al, 2013;Bawa and Kumar, 2019;Feng and Lu, 2019), played only a minor role. We also observed that the effect of each hyperparameter depends on the layer, e.g.…”

Section: Hyperparameter Importancementioning

confidence: 96%

Exploring Deep Learning for Complex Trait Genomic Prediction in Polyploid Outcrossing Species

et al. 2020

View full text Add to dashboard Cite

Section: Hyperparameter Importancementioning

confidence: 96%

Exploring Deep Learning for Complex Trait Genomic Prediction in Polyploid Outcrossing Species

et al. 2020

View full text Add to dashboard Cite

“…where x i denotes the inputs, ω i denotes the weights, Σ is the summing function, f is the activation function, and y is the output. There are three activation functions-tangent hyperbolic, sigmoid, and linear functions-in the ANNs that are used for research [31,32]. Of these, the sigmoid function is commonly applied to generalize an ANN, which predicts the probability as an output because probability exists only between 0 and 1.…”

Section: Multilayer Perceptronmentioning

confidence: 99%

Machine Learning Applied to the NOx Prediction of Diesel Vehicle under Real Driving Cycle

et al. 2021

View full text Add to dashboard Cite

Euro 6 standards impose stringent nitrogen oxide (NOx) emission limits on diesel cars. NOx emissions are significantly different between Euro 6 diesel cars and the previous standards in real-world driving. In this research, the NOx concentrations of Euro 6 diesel engines during real-world driving were studied considering various factors. Real driving emission (RDE) tests were conducted using vehicles equipped with portable emissions measurement systems. Urban, rural, and motorway test routes were utilized. Road environment, atmospheric, and after-treatment performance factors were collected in each case. An artificial neural network was used for evaluation using RDE test data and various statistical parameters. It was found that the proposed method predicted the pollutant emissions effectively. Lastly, the relative importance of each predictor was derived, and the NOx concentrations were analyzed. These approaches provide accurate emission information for an environmental effect evaluation that reflects more realistic road conditions.

show abstract

“…Another "dynamic" function is proposed in [27], whose shape is learned by a linear regression model. In [28], two different variants are proposed: a "linear sigmoidal activation", which is a fixed structure function whose function coefficients are static, and its "dynamic" variant, named "adaptive linear sigmoidal activation", which can adapt itself according to the complexity of the given data. Two of the best performing functions are Swish [29], which is the combination of a sigmoid function and a trainable parameter, and the recent Mexican ReLU (MeLU) [30], which is a piecewise linear activation function that is the sum of PReLU and multiple Mexican hat functions.…”

Section: Literature Reviewsmentioning

confidence: 99%

Stochastic Selection of Activation Layers for Convolutional Neural Networks

Nanni

Lumini

Ghidoni

et al. 2020

Sensors

View full text Add to dashboard Cite

In recent years, the field of deep learning has achieved considerable success in pattern recognition, image segmentation, and many other classification fields. There are many studies and practical applications of deep learning on images, video, or text classification. Activation functions play a crucial role in discriminative capabilities of the deep neural networks and the design of new “static” or “dynamic” activation functions is an active area of research. The main difference between “static” and “dynamic” functions is that the first class of activations considers all the neurons and layers as identical, while the second class learns parameters of the activation function independently for each layer or even each neuron. Although the “dynamic” activation functions perform better in some applications, the increased number of trainable parameters requires more computational time and can lead to overfitting. In this work, we propose a mixture of “static” and “dynamic” activation functions, which are stochastically selected at each layer. Our idea for model design is based on a method for changing some layers along the lines of different functional blocks of the best performing CNN models, with the aim of designing new models to be used as stand-alone networks or as a component of an ensemble. We propose to replace each activation layer of a CNN (usually a ReLU layer) by a different activation function stochastically drawn from a set of activation functions: in this way, the resulting CNN has a different set of activation function layers.

show abstract

Linearized sigmoidal activation: A novel activation function with tractable non-linear characteristics to boost representation capability

Cited by 45 publications

References 23 publications

Exploring Deep Learning for Complex Trait Genomic Prediction in Polyploid Outcrossing Species

Exploring Deep Learning for Complex Trait Genomic Prediction in Polyploid Outcrossing Species

Machine Learning Applied to the NOx Prediction of Diesel Vehicle under Real Driving Cycle

Stochastic Selection of Activation Layers for Convolutional Neural Networks

Contact Info

Product

Resources

About