Effects of Nonlinearity and Network Architecture on the Performance of Supervised Neural Networks

Abstract

The nonlinearity of activation functions used in deep learning models is crucial for the success of predictive models. Several simple nonlinear functions, including Rectified Linear Unit (ReLU) and Leaky-ReLU (L-ReLU) are commonly used in neural networks to impose the nonlinearity. In practice, these functions remarkably enhance the model accuracy. However, there is limited insight into the effects of nonlinearity in neural networks on their performance. Here, we investigate the performance of neural network models as a function of nonlinearity using ReLU and L-ReLU activation functions in the context of different model architectures and data domains. We use entropy as a measurement of the randomness, to quantify the effects of nonlinearity in different architecture shapes on the performance of neural networks. We show that the ReLU nonliearity is a better choice for activation function mostly when the network has sufficient number of parameters. However, we found that the image classification models with transfer learning seem to perform well with L-ReLU in fully connected layers. We show that the entropy of hidden layer outputs in neural networks can fairly represent the fluctuations in information loss as a function of nonlinearity. Furthermore, we investigate the entropy profile of shallow neural networks as a way of representing their hidden layer dynamics.

The Use of Collections of Artificial Neural Networks to Improve the Control Quality of the Induction Soldering Process

“…The supervised learning of an artificial neural network is a multidimensional optimization problem [ 17 ]. To solve this problem, it is necessary to find a set of weighting coefficients that minimize the error function.…”

Digital handwriting characteristics for dysgraphia detection using artificial neural network

“…As indicated in Figure 1, we used the python deep learning APi Keras [22] to build the ANN model, which is a simple, fast, and powerful tool for building ANN models. Activation functions are an important part of the role that ANN may play in dealing with non-linear situations [23]. The rectified linear unit (ReLU) [24] is one of the most often used activation functions in deep neural networks:…”

Wavelet-enabled Star Classification: A Deep Learning Approach for Identifying Binary and Exoplanet Stars

“…As with standard neural networks, each neuron in this layer is directly connected to all neurons in the layers below. To introduce non-linearity in the model, the ReLU activation function (Agarap 2018) (Ramachandran et al 2017) is used in this layer, as it has been shown to be effective in deep learning architectures, as discussed in (Agarap 2018) and (Kulathunga et al 2021). One of the major reasons behind the effective performance is its non-saturating…”