Accelerating the Activation Function Selection for Hybrid Deep Neural Networks – FPGA Implementation

Waseem, Shaik Mohammed; Suraj, Alavala Venkata; Roy, Subir Kumar

doi:10.1109/tensymp52854.2021.9551000

Cited by 3 publications

(1 citation statement)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While the Hard sigmoid is not as commonly used as the logistic sigmoid, it can be used, for example, in binarized neural network with stochastic activation functions [253] -the binaryized neural networks can lead to much faster inference than regular neural networks, e.g., Courbariaux et al reached up to 7× speed up without any loss in classification accuracy [253] (however, even better speed-ups can be obtained using, for example, field-programmable gate array (FPGA) implementations as in [255]).…”

Section: Hard Sigmoidmentioning

confidence: 99%

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

Kunc

Kléma²

2019

Preprint

View full text Add to dashboard Cite

Motivation: Gene expression profiling was made cheaper by the NIH LINCS program that profiles only ∼1, 000 selected landmark genes and uses them to reconstruct the whole profile. The D-GEX method employs neural networks to infer the whole profile. However, the original D-GEX can be further significantly improved. Results: We have analyzed the D-GEX method and determined that the inference can be improved using a logistic sigmoid activation function instead of the hyperbolic tangent. Moreover, we propose a novel transformative adaptive activation function that improves the gene expression inference even further and which generalizes several existing adaptive activation functions. Our improved neural network achieves average mean absolute error of 0.1340 which is a significant improvement over our reimplementation of the original D-GEX which achieves average mean absolute error 0.1637

show abstract

Section: Hard Sigmoidmentioning

confidence: 99%