First Power Linear Unit with Sign

Abstract

Convolutional neural networks (CNNs) have shown great dominance in vision tasks because of their tremendous capability to extract features from visual patterns, in which activation units serve as crucial elements for the entire structure, introducing non-linearity to the data processing systematically. This paper proposes a novel and insightful activation method termed FPLUS, which exploits mathematical power function with polar signs in form. It is enlightened by common inverse operation while endowed with an intuitive meaning of bionics. The formulation is derived theoretically under conditions of some prior knowledge and anticipative properties, and then its feasibility is verified through a series of experiments using typical benchmark datasets, whose results indicate our approach owns superior competitiveness among numerous activation functions, as well as compatible stability across many CNN architectures. Furthermore, we extend the function presented to a more generalized type called PFPLUS with two parameters that can be fixed or learnable, so as to augment its expressive capacity, and outcomes of identical tests validate this improvement. Therefore we believe the work in this paper has a certain value of enriching the family of activation units.