TanhSoft -- a family of activation functions combining Tanh and Softplus

Abstract

Deep learning at its core, contains functions that are composition of a linear transformation with a non-linear function known as activation function. In past few years, there is an increasing interest in construction of novel activation functions resulting in better learning. In this work, we propose a family of novel activation functions, namely TanhSoft, with four undetermined hyper-parameters of the form tanh (αx + βe γx ) ln(δ + e x ) and tune these hyper-parameters to obtain activation functions which are shown to outperform several well known activation functions. For instance, replacing ReLU with x tanh(0.6e x ) improves top-1 classification accuracy on CIFAR-10 by 0.46% for DenseNet-169 and 0.7% for Inception-v3 while with tanh(0.87x) ln(1 + e x ) top-1 classification accuracy on CIFAR-100 improves by 1.24% for DenseNet-169 and 2.57% for SimpleNet model.

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

“…The TanhSoft is a family of AAFs proposed in [461] that combine the softplus and tanh that contains three notable cases -TanhSoft-1, TanhSoft-2, and TanhSoft-3 [461,462].…”

“…where b i and c i are trainable parameters [461,462]. The TanhSoft-2 can be obtained from the general TanhSoft by setting a i = 0 and d i = 0 [461].…”

“…where b i and c i are trainable parameters [461,462]. The TanhSoft-2 can be obtained from the general TanhSoft by setting a i = 0 and d i = 0 [461]. The last AF from [462], TanhSoft-3, is defined as…”