The Influence of the Activation Function in a Convolution Neural Network Model of Facial Expression Recognition

Wang, Yingying; Li, Yibin; Song, Yong; Rong, Xuewen

doi:10.3390/app10051897

Cited by 233 publications

(105 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Recently, several papers related to making a more adequate activation function have appeared, mainly comparing activation functions on large datasets. To illustrate some of important contributions, the influence of the activation function in the convolutional neural network (CNN) model is studied in [24], improving a ReLU activation by construction of a novel surrogate. Theoretical analysis about gradient instability as well as the fundamental explanation for the exploding/vanishing gradient and the performances of different activation functions are given in [13].…”

Section: Introductionmentioning

confidence: 99%

“SPOCU”: scaled polynomial constant unit activation function

Kiseľák

Lü

Švihra

et al. 2020

Neural Comput & Applic

View full text Add to dashboard Cite

We address the following problem: given a set of complex images or a large database, the numerical and computational complexity and quality of approximation for neural network may drastically differ from one activation function to another. A general novel methodology, scaled polynomial constant unit activation function ''SPOCU,'' is introduced and shown to work satisfactorily on a variety of problems. Moreover, we show that SPOCU can overcome already introduced activation functions with good properties, e.g., SELU and ReLU, on generic problems. In order to explain the good properties of SPOCU, we provide several theoretical and practical motivations, including tissue growth model and memristive cellular nonlinear networks. We also provide estimation strategy for SPOCU parameters and its relation to generation of random type of Sierpinski carpet, related to the [pppq] model. One of the attractive properties of SPOCU is its genuine normalization of the output of layers. We illustrate SPOCU methodology on cancer discrimination, including mammary and prostate cancer and data from Wisconsin Diagnostic Breast Cancer dataset. Moreover, we compared SPOCU with SELU and ReLU on large dataset MNIST, which justifies usefulness of SPOCU by its very good performance.

show abstract

Section: Introductionmentioning

confidence: 99%

“SPOCU”: scaled polynomial constant unit activation function

Kiseľák

Lü

Švihra

et al. 2020

Neural Comput & Applic

View full text Add to dashboard Cite

show abstract

“…A feature map is getting by performing convolution processes to the input image or prior features using a linear filter, merging a bias term. Then passing this feature map through a non-linear activation function such as Sigmoid [34] and Rectified Linear Unit (RELU) [35] . In contrast, the classifier base includes the dense layers combined with the activation layers to convert the feature maps to one dimension vectors to expedite the classification task using many neurons.…”

Section: Theory and Methodsmentioning

confidence: 99%

An optimized deep learning architecture for the diagnosis of COVID-19 disease based on gravitational search optimization

Ezzat

Hassanien

Ella

2021

Applied Soft Computing

159

View full text Add to dashboard Cite

In this paper, a novel approach called GSA-DenseNet121-COVID-19 based on a hybrid convolutional neural network (CNN) architecture is proposed using an optimization algorithm. The CNN architecture that was used is called DenseNet121, and the optimization algorithm that was used is called the gravitational search algorithm (GSA). The GSA is used to determine the best values for the hyperparameters of the DenseNet121 architecture. To help this architecture to achieve a high level of accuracy in diagnosing COVID-19 through chest x-ray images. The obtained results showed that the proposed approach could classify 98.38% of the test set correctly. To test the efficacy of the GSA in setting the optimum values for the hyperparameters of DenseNet121. The GSA was compared to another approach called SSD-DenseNet121, which depends on the DenseNet121 and the optimization algorithm called social ski driver (SSD). The comparison results demonstrated the efficacy of the proposed GSA-DenseNet121-COVID-19. As it was able to diagnose COVID-19 better than SSD-DenseNet121 as the second was able to diagnose only 94% of the test set. The proposed approach was also compared to another method based on a CNN architecture called Inception-v3 and manual search to quantify hyperparameter values. The comparison results showed that the GSA-DenseNet121-COVID-19 was able to beat the comparison method, as the second was able to classify only 95% of the test set samples. The proposed GSA-DenseNet121-COVID-19 was also compared with some related work. The comparison results showed that GSA-DenseNet121-COVID-19 is very competitive.

show abstract

“…The current common activation functions include Sigmoid, TanHyperbolic (tanh), and Rectified Linear Unit (ReLU) functions. The ReLU function can greatly improve the performance of CNN, and its performance in training speed is also better than that of other functions [30]. The ReLU function is expressed as (15).…”

Section: Convolutional Neural Network Work (Cnn)mentioning

confidence: 99%

Fault Diagnosis for Power Cables Based on Convolutional Neural Network With Chaotic System and Discrete Wavelet Transform

Wang

Liao

2022

IEEE Trans. Power Delivery

View full text Add to dashboard Cite

In this paper, the discrete wavelet transform (DWT) and a chaotic system were combined with a convolutional neural network (CNN) and applied to the diagnosis of insulation faults in XLPE (cross-linked polyacetylene) power cables. First, four different types of insulation faults in power cables were constructed, including the normal state of the cable, the short outer semi-conducting layer, impurities in the insulation layer, and insulation layer damage, and a high-speed capture card (NI PXI-5105) was adopted to measure the partial discharge (PD) signal, which was then filtered through discrete wavelet transform. Then, based on the Lorenz chaotic system, a dynamic error scatter diagram was established as the feature of each fault state. Finally, the dynamic error scatter diagram was processed by CNN to recognize four different types of faults in the power cable. The test results show that the method proposed in this paper can quickly recognize the fault state of power cables and has excellent performance in terms of recognition accuracy, which reaches 97.5%. Therefore, the proposed method can effectively detect the fault signal changes of power cables and identify the fault state of power cables in real time.

show abstract

The Influence of the Activation Function in a Convolution Neural Network Model of Facial Expression Recognition

Cited by 233 publications

References 34 publications

“SPOCU”: scaled polynomial constant unit activation function

“SPOCU”: scaled polynomial constant unit activation function

An optimized deep learning architecture for the diagnosis of COVID-19 disease based on gravitational search optimization

Fault Diagnosis for Power Cables Based on Convolutional Neural Network With Chaotic System and Discrete Wavelet Transform

Contact Info

Product

Resources

About