A comparative performance analysis of different activation functions in LSTM networks for classification

Farzad, Amir; Mashayekhi, Hoda; Hassanpour, Hamid

doi:10.1007/s00521-017-3210-6

Cited by 108 publications

(69 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In addition, time-sequential can be stored through the recurrent weights of the network, and recurrent neurons can then reflect time sequences. Therefore, an RNN could estimate disturbances better than a conventional feedforward techniques [48,67].…”

Section: A Selecting An Rnn-based Controllermentioning

confidence: 99%

Recurrent Neural Network-Based Robust Nonsingular Sliding Mode Control With Input Saturation for a Non-Holonomic Spherical Robot

et al. 2020

View full text Add to dashboard Cite

We develop a new robust control scheme for a non-holonomic spherical robot. To this end, the mathematical model of a pendulum driven non-holonomic spherical robot is first presented. Then, a recurrent neural network-based robust nonsingular sliding mode control is proposed for stabilization and tracking control of the system. The designed recurrent neural network is applied to approximate compound disturbances, including external interferences and dynamic uncertainties. Moreover, the controller is designed in a way that avoids the singularity problem in the system. Another advantage of the proposed scheme is its ability for tracking control while there exists control input saturation, which is a serious concern in robotic systems. Based on the Lyapunov theorem, the stability of the closed-loop system has also been confirmed. Lastly, the performance of the proposed control technique for the uncertain system in the presence of an external disturbance, unknown input saturation, and dynamic uncertainties has been investigated. Also, the proposed controller has been compared with a Fuzzy-PID one. Simulation results show the effectiveness and superiority of the developed control technique.

show abstract

Section: A Selecting An Rnn-based Controllermentioning

confidence: 99%

Recurrent Neural Network-Based Robust Nonsingular Sliding Mode Control With Input Saturation for a Non-Holonomic Spherical Robot

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Because of the data passing through these non-linear transformations within the memory cell, it is common practise to not include any further activation at the nodes of the hidden layers. There is, however, discussion in the literature about choosing different activation functions for RNNs, for example Farzad et al (2019) investigate alternatives to the sigmoid activations at the LSTM input, forget and output gates. We add an activation at the final output layer only, and as our task is regression, we use a linear activation here as we do in the FFNN.…”

Section: Simulating Groundwater Levels-simple Modelmentioning

confidence: 99%

Modern Strategies for Time Series Regression

Clark

Hyndman

Pagendam

et al. 2020

Int Statistical Rev

View full text Add to dashboard Cite

This paper discusses several modern approaches to regression analysis involving time series data where some of the predictor variables are also indexed by time. We discuss classical statistical approaches as well as methods that have been proposed recently in the machine learning literature. The approaches are compared and contrasted, and it will be seen that there are advantages and disadvantages to most currently available approaches. There is ample room for methodological developments in this area. The work is motivated by an application involving the prediction of water levels as a function of rainfall and other climate variables in an aquifer in eastern Australia.

show abstract

“…However, it is more expensive to compute than tanh; in other words, it has more complex derivatives. Additionally, its gradient sometimes yields extremely low/high values, such that we can consider it as a sigmoid on steroids [26]- [28].…”

Section: Softsign Activation Functionmentioning

confidence: 99%

Review and Comparison of Commonly Used Activation Functions for Deep Neural Networks

Szandała

2020

Studies in Computational Intelligence

238

View full text Add to dashboard Cite

The primary neural networks decision-making units are activation functions. Moreover, they evaluate the output of networks neural node; thus, they are essential for the performance of the whole network. Hence, it is critical to choose the most appropriate activation function in neural networks calculation. Acharya et al. (2018) suggest that numerous recipes have been formulated over the years, though some of them are considered deprecated these days since they are unable to operate properly under some conditions. These functions have a variety of characteristics, which are deemed essential to successfully learning. Their monotonicity, individual derivatives, and finite of their range are some of these characteristics (Bach 2017). This research paper will evaluate the commonly used additive functions, such as swish, ReLU, Sigmoid, and so forth. This will be followed by their properties, own cons and pros, and particular formula application recommendations.

show abstract

A comparative performance analysis of different activation functions in LSTM networks for classification

Cited by 108 publications

References 35 publications

Recurrent Neural Network-Based Robust Nonsingular Sliding Mode Control With Input Saturation for a Non-Holonomic Spherical Robot

Recurrent Neural Network-Based Robust Nonsingular Sliding Mode Control With Input Saturation for a Non-Holonomic Spherical Robot

Modern Strategies for Time Series Regression

Review and Comparison of Commonly Used Activation Functions for Deep Neural Networks

Contact Info

Product

Resources

About