TanhExp: A smooth activation function with high convergence speed for lightweight neural networks

Liu, Xinyu; Di, Xiaoguang

doi:10.1049/cvi2.12020

Cited by 39 publications

(18 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In a recent study by Liu et al [24], the authors proposed the Tanh exponential activation function (TanhExp), which improves the performance of lightweight or mobile neural networks used for real-time computer vision tasks, and contains fewer parameters than usual. It enhances the performance of these networks on image classification.…”

Section: Related Workmentioning

confidence: 99%

SinLU: Sinu-Sigmoidal Linear Unit

et al. 2022

View full text Add to dashboard Cite

Non-linear activation functions are integral parts of deep neural architectures. Given the large and complex dataset of a neural network, its computational complexity and approximation capability can differ significantly based on what activation function is used. Parameterizing an activation function with the introduction of learnable parameters generally improves the performance. Herein, a novel activation function called Sinu-sigmoidal Linear Unit (or SinLU) is proposed. SinLU is formulated as SinLU(x)=(x+asinbx)·σ(x), where σ(x) is the sigmoid function. The proposed function incorporates the sine wave, allowing new functionalities over traditional linear unit activations. Two trainable parameters of this function control the participation of the sinusoidal nature in the function, and help to achieve an easily trainable, and fast converging function. The performance of the proposed SinLU is compared against widely used activation functions, such as ReLU, GELU and SiLU. We showed the robustness of the proposed activation function by conducting experiments in a wide array of domains, using multiple types of neural network-based models on some standard datasets. The use of sine wave with trainable parameters results in a better performance of SinLU than commonly used activation functions.

show abstract

Section: Related Workmentioning

confidence: 99%

SinLU: Sinu-Sigmoidal Linear Unit

et al. 2022

View full text Add to dashboard Cite

show abstract

“…The author of [26] has reported unstable training behaviour for a specific function which can be obtained from TanhSoft-1, however, we tested and failed to find any such instability. Also, in [31] the authors have mentioned a special case which can be obtained from TanhSoft-2.…”

Section: Tanhsoft-1 Tanhsoft-2 and Tanhsoft-3 And Their Propertiesmentioning

confidence: 99%

TanhSoft—Dynamic Trainable Activation Functions for Faster Learning and Better Performance

et al. 2021

View full text Add to dashboard Cite

Deep learning, at its core, contains functions that are the composition of a linear transformation with a non-linear function known as activation function. In the past few years, there is an increasing interest in the construction of novel activation functions resulting in better learning. In this work, we propose three novel activation functions with learnable parameters, namely TanhSoft-1, TanhSoft-2, and TanhSoft-3, which are shown to outperform several well-known activation functions. For instance, replacing ReLU with TanhSoft-1, TanhSoft-2, and Tanhsot-3 improves top-1 classification accuracy by 6.06%, 5.75%, and 5.38% respectively on VGG-16(with batch-normalization), by 3.02%, 3.25% and 2.93% respectively on PreActResNet-34 in CIFAR-100 dataset, by 1.76%, 1.93%, and 1.82% respectively on WideResNet 28-10 in Tiny ImageNet dataset. TanhSoft-1, TanhSoft-2, and Tanhsot-3 outperformed ReLU on mean average precision (mAP) by 0.7%, 0.8%, and 0.6% respectively in object detection problem on SSD 300 model in Pascal VOC dataset.

show abstract

“…Worth noting that the author has reported unstable training behaviour for the specific function f (x; 1, 0, γ, 1) in [20], however, we failed to find any instability during the training process. Also, in [21] the authors have mentioned the function f (x; 0, 1, 1, 0), which arise as an example from the TanhSoft family. In fact, we show that because of the introduction of hyper-parameters, better activation functions of the form f (x; 0, β, γ, 0) can be obtained.…”

Section: Tanhsoft Activation Function Familymentioning

confidence: 99%

TanhSoft -- a family of activation functions combining Tanh and Softplus

Biswas¹,

Kumar²,

Banerjee³

et al. 2020

Preprint

View full text Add to dashboard Cite

Deep learning at its core, contains functions that are composition of a linear transformation with a non-linear function known as activation function. In past few years, there is an increasing interest in construction of novel activation functions resulting in better learning. In this work, we propose a family of novel activation functions, namely TanhSoft, with four undetermined hyper-parameters of the form tanh (αx + βe γx ) ln(δ + e x ) and tune these hyper-parameters to obtain activation functions which are shown to outperform several well known activation functions. For instance, replacing ReLU with x tanh(0.6e x ) improves top-1 classification accuracy on CIFAR-10 by 0.46% for DenseNet-169 and 0.7% for Inception-v3 while with tanh(0.87x) ln(1 + e x ) top-1 classification accuracy on CIFAR-100 improves by 1.24% for DenseNet-169 and 2.57% for SimpleNet model.

show abstract

TanhExp: A smooth activation function with high convergence speed for lightweight neural networks

Cited by 39 publications

References 19 publications

SinLU: Sinu-Sigmoidal Linear Unit

SinLU: Sinu-Sigmoidal Linear Unit

TanhSoft—Dynamic Trainable Activation Functions for Faster Learning and Better Performance

TanhSoft -- a family of activation functions combining Tanh and Softplus

Contact Info

Product

Resources

About