TanhSoft—Dynamic Trainable Activation Functions for Faster Learning and Better Performance

Biswas, Koushik; Kumar, Sandeep; Banerjee, Shilpak; Pandey, Ashish Kumar

doi:10.1109/access.2021.3105355

Cited by 14 publications

(4 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The TanhSoft is a family of AAFs proposed in [461] that combine the softplus and tanh that contains three notable cases -TanhSoft-1, TanhSoft-2, and TanhSoft-3 [461,462].…”

Section: Tanhsoftmentioning

confidence: 99%

“…where a i is a trainable parameter [461,462]; it can be obtained from the general TanhSoft by setting b i = 0 and d i = 1 [461]. The second AF from [462], TanhSoft-2, is defined as…”

Section: Tanhsoftmentioning

confidence: 99%

“…where b i and c i are trainable parameters [461,462]. The TanhSoft-2 can be obtained from the general TanhSoft by setting a i = 0 and d i = 0 [461].…”

Section: Tanhsoftmentioning

confidence: 99%

See 2 more Smart Citations

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

Kunc

Kléma²

2019

Preprint

View full text Add to dashboard Cite

Motivation: Gene expression profiling was made cheaper by the NIH LINCS program that profiles only ∼1, 000 selected landmark genes and uses them to reconstruct the whole profile. The D-GEX method employs neural networks to infer the whole profile. However, the original D-GEX can be further significantly improved. Results: We have analyzed the D-GEX method and determined that the inference can be improved using a logistic sigmoid activation function instead of the hyperbolic tangent. Moreover, we propose a novel transformative adaptive activation function that improves the gene expression inference even further and which generalizes several existing adaptive activation functions. Our improved neural network achieves average mean absolute error of 0.1340 which is a significant improvement over our reimplementation of the original D-GEX which achieves average mean absolute error 0.1637

show abstract

“…The TanhSoft is a family of AAFs proposed in [461] that combine the softplus and tanh that contains three notable cases -TanhSoft-1, TanhSoft-2, and TanhSoft-3 [461,462].…”

Section: Tanhsoftmentioning

confidence: 99%

“…where a i is a trainable parameter [461,462]; it can be obtained from the general TanhSoft by setting b i = 0 and d i = 1 [461]. The second AF from [462], TanhSoft-2, is defined as…”

Section: Tanhsoftmentioning

confidence: 99%

See 1 more Smart Citation

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

Kunc

Kléma²

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Venkatappareddy et al [38] proposed a polynomial-based activation function; Hegui et al [39] proposed a nonmonotonic activation function called Logish; both show high accuracy on the CIFAR-10 dataset. TanhSoft-1, TanhSoft-2, and TanhSoft-3 [40] are combinations of Tanh; the exponential terms also increase the computational complexity. ALKHOULY et al [41] proposed the IpLU and AbsLU without exponential terms to make it simple in computation.…”

Section: Introductionmentioning

confidence: 99%

PolyLU: A Simple and Robust Polynomial-Based Linear Unit Activation Function for Deep Learning

Feng,

Yang

2023

IEEE Access

View full text Add to dashboard Cite

The activation function has a critical influence on whether a convolutional neural network in deep learning can converge or not; a proper activation function not only makes the convolutional neural network converge faster but also can reduce the complexity of convolutional neural network architecture and gets the same or better performance. Many activation functions have been proposed; however, various activation functions have advantages, defects, and applicable network architectures. A new activation function called Polynomial Linear Unit (PolyLU) is proposed in this paper to improve some of the shortcomings of the existing activation functions. The PolyLU meets the following basic properties: continuously differentiable, approximate identity near the origin, unbounded for positive inputs, bounded for negative inputs, smooth, monotonic, and zero-centered. There is a polynomial term for the negative inputs and no exponential terms in the PolyLU that reduces the computational complexity of the network. Compared to those common activation functions like Sigmoid, Tanh, ReLU, LeakyReLU, ELU, Mish, and Swish, the experiments show that the PolyLU has improved some network complexity and has better accuracy over MNIST, Kaggle Cats and Dogs, CIFAR-10 and CIFAR-100 datasets. Test by the CIFAR-100 dataset with batch normalization, PolyLU improves by 0.

show abstract

Residual Wavelon Convolutional Networks for Characterization of Disease Response on MRI

Sadri¹,

DeSilvio²,

Chirra³

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

TanhSoft—Dynamic Trainable Activation Functions for Faster Learning and Better Performance

Cited by 14 publications

References 30 publications

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

PolyLU: A Simple and Robust Polynomial-Based Linear Unit Activation Function for Deep Learning

Residual Wavelon Convolutional Networks for Characterization of Disease Response on MRI

Contact Info

Product

Resources

About