Universal activation function for machine learning

Yuen, Brosnan; Hoang, Minh Tu; Dong, Xiaodai; Lü, Tao

doi:10.1038/s41598-021-96723-8

Cited by 42 publications

(22 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We used sigmoid as an activation function. The main reason why we use the sigmoid function is that the proposed model predicts probability as an output [ 23 ].…”

Section: Methodsmentioning

confidence: 99%

Bi-directional long short term memory-gated recurrent unit model for Amharic next word prediction

2022

View full text Add to dashboard Cite

The next word prediction is useful for the users and helps them to write more accurately and quickly. Next word prediction is vital for the Amharic Language since different characters can be written by pressing the same consonants along with different vowels, combinations of vowels, and special keys. As a result, we present a Bi-directional Long Short Term-Gated Recurrent Unit (BLST-GRU) network model for the prediction of the next word for the Amharic Language. We evaluate the proposed network model with 63,300 Amharic sentence and produces 78.6% accuracy. In addition, we have compared the proposed model with state-of-the-art models such as LSTM, GRU, and BLSTM. The experimental result shows, that the proposed network model produces a promising result.

show abstract

“…We used sigmoid as an activation function. The main reason why we use the sigmoid function is that the proposed model predicts probability as an output [ 23 ].…”

Section: Methodsmentioning

confidence: 99%

Bi-directional long short term memory-gated recurrent unit model for Amharic next word prediction

2022

View full text Add to dashboard Cite

show abstract

“…The precision value is above 0.8 for all activation functions except for the ReLu activation function (0.01), the highest precision value is achieved by the soft plus and UAF activation functions with a value of 0.902, the recall value is above 0.8 for all activation functions except the activation function. ReLu (0.1), the highest recall value was achieved by the softplus activation function and UAF with a value of 0.902, the F1 value was above 0.8 for all activation functions except the ReLu activation function (0.018) [9].…”

Section: Introductionmentioning

confidence: 91%

Comparison of Multilayer Perceptron’s Activation and Op-Timization Functions in Classification of Covid-19 Patients

Pardede¹,

Firmansyah²,

Handayani³

et al. 2022

JURTEKSI

View full text Add to dashboard Cite

Patient’s symptoms could be used as features in Covid-19 classification. Using multi layer perceptron, the classification uses data set that contains patient’s diagnosis which has Covid-19 symptoms dan processes the data set to see if the patient is Covid-19 positive or not. This paper compare four activation function such as identity, logistic, ReLu and tanh and combined them with optimizer such as L-BFGS-B, SGD and Adam. Using 5-fold and 10-fold cross validation technique to get the accuracy, F1, precision and recall values, the result that we get is that logistic function with L-BFGS-B optimizer and ReLu function with L-BFGS-B optimizer are the best combinations. The logistic function with SGD optimizer, ReLu function with Adam optimizer and tanh function with Adam optimizer are the worst combinations according to their accuration values. The logistic function with SGD optimizer is the worst combination according to its F1 value. The logistic function with SGD optimizer and tanh function with L-BFGS-B optimizer are the worst combinations according to their precision values. The logistic function with SGD optimizer, ReLu function with Adam optimizer and tanh function with Adam optimizer are the worst combinations according to their recall values. Keywords: activation function, covid-19; multi layer perceptron; optimizer algorithm Abstrak: Diagnosa gejala yang dialami pasien dapat digunakan sebagai fitur dalam klasifikasi penderita Covid-19. Dengan multi layer perceptron, klasifikasi dilakukan menggunakan data set yang berisi hasil diagnosa pasien yang memiliki gejala Covid-19 dan selanjutnya diolah untuk melihat apakah memang pasien tersebut menderita Covid-19 atau tidak. Penelitian ini membandingkan fungsi aktivasi identity, logistic, ReLu dan tanh yang dikombinasikan dengan algoritma optimasi L-BFGS-B, SGD dan Adam. Hasil evaluasi cross validation menggunakan 5-fold dan 10-fold digunakan sebagai dasar menentukan kombinasi yang terbaik dan terburuk, dengan hasil yang menunjukkan bahwa kombinasi fungsi logistic dengan optimasi L-BFGS-B dan fungsi ReLu dengan optimasi L-BFGS-B merupakan kombinasi terbaik. Kombinasi fungsi logisctic dengan optimasi SGD, fungsi ReLu dengan optimasi Adam dan fungsi tanh dengan optimasi Adam merupakan yang terburuk dari nilai accuracy. Kombinasi fungsi logistic dan optimasi SGD merupakan kombinasi terburuk dari nilai F1. Kombinasi fungsi logistic dengan optimasi SGD dan fungsi tanh dan optimasi L-BFGS-B merupakan yang terburuk dari nilai precision. Kombinasi fungsi logisctic dengan optimasi SGD, fungsi ReLu dengan optimasi Adam dan fungsi tanh dengan optimasi Adam merupakan kombinasi terburuk dari nilai recall. Kata kunci: algoritma optimasi; covid-19; fungsi optimasi; multi layer perceptron

show abstract

“…For example, two-section distributed-feedback (DFB) lasers 20 , vertical-cavity surface-emitting laser (VCSEL) 21 and disk lasers 22 have shown promising results, but they are bottlenecked by network scale, frequency of access and power consumption. Moreover, their nonlinear activation responses tend to be fixed during accelerator fabrication, but the nonlinear activation forms should be reprogrammed according to different ANN models and data sets 23 . Thus, as a complementary approach, a more straightforward and flexible implementation is attained by calculating the nonlinear functions in CPU, which connects physical photonic neural networks through electrical-to-optical (E/O) and optical-to-electrical (O/E) converters 24,25 .…”

Section: Introductionmentioning

confidence: 99%

Reconfigurable nonlinear photonic activation function for photonic neural network based on non-volatile opto-resistive RAM switch

Tang

Zhang

et al. 2022

Light Sci Appl

View full text Add to dashboard Cite

Photonic neural network has been sought as an alternative solution to surpass the efficiency and speed bottlenecks of electronic neural network. Despite that the integrated Mach–Zehnder Interferometer (MZI) mesh can perform vector-matrix multiplication in photonic neural network, a programmable in-situ nonlinear activation function has not been proposed to date, suppressing further advancement of photonic neural network. Here, we demonstrate an efficient in-situ nonlinear accelerator comprising a unique solution-processed two-dimensional (2D) MoS2 Opto-Resistive RAM Switch (ORS), which exhibits tunable nonlinear resistance switching that allow us to introduce nonlinearity to the photonic neuron which overcomes the linear voltage-power relationship of typical photonic components. Our reconfigurable scheme enables implementation of a wide variety of nonlinear responses. Furthermore, we confirm its feasibility and capability for MNIST handwritten digit recognition, achieving a high accuracy of 91.6%. Our accelerator constitutes a major step towards the realization of in-situ photonic neural network and pave the way for the integration of photonic integrated circuits (PIC).

show abstract

Universal activation function for machine learning

Cited by 42 publications

References 22 publications

Bi-directional long short term memory-gated recurrent unit model for Amharic next word prediction

Bi-directional long short term memory-gated recurrent unit model for Amharic next word prediction

Comparison of Multilayer Perceptron’s Activation and Op-Timization Functions in Classification of Covid-19 Patients

Reconfigurable nonlinear photonic activation function for photonic neural network based on non-volatile opto-resistive RAM switch

Contact Info

Product

Resources

About