“SPOCU”: scaled polynomial constant unit activation function

Abstract

We address the following problem: given a set of complex images or a large database, the numerical and computational complexity and quality of approximation for neural network may drastically differ from one activation function to another. A general novel methodology, scaled polynomial constant unit activation function ''SPOCU,'' is introduced and shown to work satisfactorily on a variety of problems. Moreover, we show that SPOCU can overcome already introduced activation functions with good properties, e.g., SELU and ReLU, on generic problems. In order to explain the good properties of SPOCU, we provide several theoretical and practical motivations, including tissue growth model and memristive cellular nonlinear networks. We also provide estimation strategy for SPOCU parameters and its relation to generation of random type of Sierpinski carpet, related to the [pppq] model. One of the attractive properties of SPOCU is its genuine normalization of the output of layers. We illustrate SPOCU methodology on cancer discrimination, including mammary and prostate cancer and data from Wisconsin Diagnostic Breast Cancer dataset. Moreover, we compared SPOCU with SELU and ReLU on large dataset MNIST, which justifies usefulness of SPOCU by its very good performance.

Correction to: “SPOCU”: scaled polynomial constant unit activation function

“…Following the publication of our article [1], we have become aware that the legends of Fig. 8b should read as follows: generator h(Á) of the activation function S. Moreover, the following Fig.…”

“…!). Shortly after publication of our article [1], we have received questions from the community how to implement SPOCU, since it is not involved in current software packages, e.g. in Matlab or Python (Keras).…”

“…]: = P. For proper choice of the parameters, it is sufficient they belong to P and they can satisfy several conditions. One set of conditions has been outlined in [1], which gives hold. Here f(z) is the pdf of underlying distribution, J is the Jacobi matrix w.r.t to mean and variance parameters and ||Á|| is the appropriate matrix norm, e.g.…”

“…E.g. one can, similarly as in 5.1. of [1] fix c = c = 1. And, to compute a, b one can consider Pareto density…”

Leveraging the Bhattacharyya coefficient for uncertainty quantification in deep neural networks

“…More recent activation functions, such as the scaled polynomial constant unit activation function (SPOCU)[22], could potentially result in a higher classification accuracy. However, for the application of uncertainty quantification, we found that ReLU activation function achieves similar results.…”