The Cramming, Softening and Integrating Learning Algorithm with Parametric ReLU Activation Function for Binary Input/Output Problems

Tsai, Yu-Han; Jheng, Yu-Jie; Tsaih, Rua‐Huan

doi:10.1109/ijcnn.2019.8852023

Cited by 3 publications

(3 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From 2013 to 2021, studies were conducted to improve an activation function for the occurrence of "necrosis" caused by ReLu. Lu et al proposed the use of randomized asymmetric weight initialization, leaky ReLus, ELU, and so on [22,23], which allow learning the negative slope, in contrast to leaky ReLU and tanh-ReLu [24,25]. Dan Hendrycks et al introduced the concept of random regularity into the Gaussian error linear unit (GELU).…”

Section: Mishmentioning

confidence: 99%

Smish: A Novel Activation Function for Deep Learning Methods

2022

View full text Add to dashboard Cite

Activation functions are crucial in deep learning networks, given that the nonlinear ability of activation functions endows deep neural networks with real artificial intelligence. Nonlinear nonmonotonic activation functions, such as rectified linear units, Tan hyperbolic (tanh), Sigmoid, Swish, Mish, and Logish, perform well in deep learning models; however, only a few of them are widely used in mostly all applications due to their existing inconsistencies. Inspired by the MB-C-BSIF method, this study proposes Smish, a novel nonlinear activation function, expressed as f(x)=x·tanh[ln(1+sigmoid(x))], which could overcome other activation functions with good properties. Logarithmic operations are first used to reduce the range of sigmoid(x). The value is then calculated using the tanh operator. Inputs are ultimately used to multiply the previous value, thus exhibiting negative output regularization. Experiments show that Smish tends to operate more efficiently than Logish, Mish, and other activation functions on EfficientNet models with open datasets. Moreover, we evaluated the performance of Smish in various deep learning models and the parameters of its function f(x)=αx·tanh[ln(1+sigmoid(βx))], and where α = 1 and β = 1, Smish was found to exhibit the highest accuracy. The experimental results show that with Smish, the EfficientNetB3 network exhibits a Top-1 accuracy of 84.1% on the CIFAR-10 dataset; the EfficientNetB5 network has a Top-1 accuracy of 99.89% on the MNIST dataset; and the EfficientnetB7 network has a Top-1 accuracy of 91.14% on the SVHN dataset. These values are superior to those obtained using other state-of-the-art activation functions, which shows that Smish is more suitable for complex deep learning models.

show abstract

Section: Mishmentioning

confidence: 99%

Smish: A Novel Activation Function for Deep Learning Methods

2022

View full text Add to dashboard Cite

show abstract

“…• 4'th step 1. The activation function φ(z) is selected and must be continuous, e.g type: ReLU [27]. 2.…”

Section: Algorithm Machine Learningmentioning

confidence: 99%

The V2G Process With the Predictive Model

Mroczek

Kolodynska

2020

IEEE Access

View full text Add to dashboard Cite

The paper proposes using a predictive model to optimize the use of electricity in the V2G (vehicle to grid) service. The novelty of the mechanism as a kind of model predictive control (MPC) is that it seeks an effective way of managing electric energy in an Electric Vehicle (EV). Additionally, it proposes a new method of predicting the electricity consumption which allows the battery of an electric vehicle to reconcile two sides: both the system's and the user's demand will be met at the same time. The model allows for very precise determination of the vehicle's demand for the energy related to the progressive movement, taking into account the parameters characteristic of a given vehicle model, its suspension structure and aerodynamics. In addition, the machine learning algorithm was proposed for the prediction model as a hybrid (offline and online) of supervised learning. As the first part of the research, by using Matlab/Simulink/dSpace software, a prediction of EV energy consumption was created on a selected route at different times of the day (offline data matrix). At the same time, the simulated route was travelled by a BMW i3 EV (online data matrix). Based on the developed machine learning algorithm the results of the electric energy consumption were compared. The research confirms that if the correct mechanism for prediction of energy consumption by the EV is used, it is possible to define the amount of energy needed for a V2G service. The measurement error was obtained at 0.5%. The added value is setting up the EV energy security of customers after the V2G service and a correct WIN-WIN relation between the Low Voltage grid and EV customers' needs.

show abstract

“…Between 2013 and 2015, some researchers have proposed improved activation functions based on the phenomenon of "necrosis" that brought by ReLu function, such as: leaky ReLus [27], ELU [28], PReLu [29], tanh-ReLu [30], and so on.…”

Section: Introductionmentioning

confidence: 99%

The Influence of the Activation Function in a Convolution Neural Network Model of Facial Expression Recognition

et al. 2020

View full text Add to dashboard Cite

The convolutional neural network (CNN) has been widely used in image recognition field due to its good performance. This paper proposes a facial expression recognition method based on the CNN model. Regarding the complexity of the hierarchic structure of the CNN model, the activation function is its core, because the nonlinear ability of the activation function really makes the deep neural network have authentic artificial intelligence. Among common activation functions, the ReLu function is one of the best of them, but it also has some shortcomings. Since the derivative of the ReLu function is always zero when the input value is negative, it is likely to appear as the phenomenon of neuronal necrosis. In order to solve the above problem, the influence of the activation function in the CNN model is studied in this paper. According to the design principle of the activation function in CNN model, a new piecewise activation function is proposed. Five common activation functions (i.e., sigmoid, tanh, ReLu, leaky ReLus and softplus–ReLu, plus the new activation function) have been analysed and compared in facial expression recognition tasks based on the Keras framework. The Experimental results on two public facial expression databases (i.e., JAFFE and FER2013) show that the convolutional neural network based on the improved activation function has a better performance than most-of-the-art activation functions.

show abstract

The Cramming, Softening and Integrating Learning Algorithm with Parametric ReLU Activation Function for Binary Input/Output Problems

Cited by 3 publications

References 23 publications

Smish: A Novel Activation Function for Deep Learning Methods

Smish: A Novel Activation Function for Deep Learning Methods

The V2G Process With the Predictive Model

The Influence of the Activation Function in a Convolution Neural Network Model of Facial Expression Recognition

Contact Info

Product

Resources

About