Comparison of ReLU and linear saturated activation functions in neural network for universal approximation

Abstract

Activation functions used in hidden layers directly affect the possibilities for describing nonlinear systems using a feedforward neural network. Furthermore, linear based activation functions are less computationally demanding than their nonlinear alternatives. In addition, feedforward neural networks with linear based activation functions can be advantageously used for control of nonlinear systems, as shown in previous authors' publications. This paper aims to compare two types of linear based functions-symmetric linear saturated function and the rectifier linear unit (ReLU) function as activation functions of the feedforward neural network used for a nonlinear system approximation. Topologies with one hidden layer and the combination of defined quantities of hidden layer neurons in the feedforward neural network are used. Strict criteria are applied for the conditions of the experiments; specifically, the Levenberg-Marquardt algorithm is applied as a training algorithm and the Nguyen-Widrow algorithm is used for the weights and biases initialization. Three benchmark systems are then selected as nonlinear plants for approximation, which should serve as a repeatable source of data for testing. The training data are acquired by the computation of the output as a reaction to a specified colored input signal. The comparison is based on the convergence speed of the training for a fixed value of the error function, and also on the performance over a constant number of epochs. At the end of the experiments, only small differences between the performance of both applied activation functions are observed. Although the symmetric linear saturated activation function provides the lesser median of the final error function value across the all tested numbers of neurons in topologies, the ReLU function seems to be also capable of use as the activation function for nonlinear system modeling.

Artificial Neural Network Based Optimal Feedforward Torque Control of Interior Permanent Magnet Synchronous Machines: A Feasibility Study and Comparison with the State-of-the-Art

“…Regarding the layers, it is stated in [65] that a single hidden layer is sufficient to train the ANN to match "any non-linearity". Following this statement, a network with one hidden layer is chosen as the minimum requirement and, for comparison, a network with two layers is implemented as well (the maximum depth of the hidden layers considered here).…”

Design Space Exploration of a Multi-Model AI-Based Indoor Localization System

“…We used the Rectified Linear Unit (ReLU) activation function for all hidden layers. It is the default activation function for many types of neural networks, because a model that uses it is easier to train and often achieves better performance [27]. For the single-neuron output layer, we used the Linear Activation (LA) function.…”

“…Since last experiment already resulted in a significantly larger model (requiring around 6 Mbytes of memory) and before continuing exploring more complex NN architectures, we noted that there still could be room for improvement without adding extra layers. Considering the fact that when working with several layers with ReLU activation we have a significant risk of dying neurons harming our performance [27], this in turn can lead to underfitting. Batch normalization (BN) is one of the best ways to handle this issue [33].…”

A New Convolutional Neural Network-Based System for NILM Applications

“…A bias value is added to this sum. The application of the AF specifically introduces nonlinearities, aiming to emulate the way humans analyze real-world data [25]. The connection between one node and another is performed by a number (weight), which can be positive (one node excites another) or negative (one node inhibits another).…”