Smooth Function Approximation by Deep Neural Networks with General Activation Functions

Abstract

There has been a growing interest in expressivity of deep neural networks. However, most of the existing work about this topic focuses only on the specific activation function such as ReLU or sigmoid. In this paper, we investigate the approximation ability of deep neural networks with a broad class of activation functions. This class of activation functions includes most of frequently used activation functions. We derive the required depth, width and sparsity of a deep neural network to approximate any Hölder smooth function upto a given approximation error for the large class of activation functions. Based on our approximation error analysis, we derive the minimax optimality of the deep neural network estimators with the general activation functions in both regression and classification problems. on the expressivity of deep neural networks, i.e., ability to approximate a rich class of functions efficiently. The well-known classical result on this topic is the universal approximation theorem, which states that every continuous function can be approximated arbitrarily well by a neural network [11,15,12,5,20]. But these results do not specify the required numbers of layers and nodes of a neural network to achieve a given approximation accuracy.Recently, several results about the effects of the numbers of layers and nodes of a deep neural network to its expressivity have been reported. They provide upper bounds of the numbers of layers and nodes required for neural networks to uniformly approximate all functions of interest. Examples of a class of functions include the space of rational functions of polynomials [30], the Hölder space [33,27,2,21], Besov and mixed Besov spaces [29] and even a class of discontinuous functions [25,16].The nonlinear activation function is a central part that makes neural networks differ from the linear models, that is, a neural network becomes a linear function if the linear activation function is used. Therefore, the choice of an activation function substantially influences on the performance and computational efficiency. Numerous activation functions have been suggested to improve neural network learning [3,6,4,26,18,32]. We refer to the papers [13,26] for an overview of this topic.There are also many recent theoretical studies about the approximation ability of deep neural networks. However, most of the studies focus on a specific type of the activation function such as ReLU [33,27,25,16,29], or small classes of activation functions such as sigmoidal functions with additional monotonicity, continuity, and/or boundedness conditions [24,9,8,10,7] and madmissible functions which are sufficiently smooth and bounded [2]. For definitions of sigmoidal and m-admissible functions, see [9] and [2], respectively. Thus a unified theoretical framework still lacks.In this paper, we investigate the approximation ability of deep neural networks with a quite general class of activation functions. We derive the required numbers of layers and nodes of a deep neural network to approximate any Hölder smo...

Deep Local Volatility

“…However, Theorem 4.1 in Ohn and Kim (2019), which is stated for the reader's convenience in Appendix B, provides support for our observation, presented in Section 5, that sparsening the network (i.e., splitting) increases the approximation error. Hence, in what follows, we also consider the so called soft constraints approach using a fully connected network, where the static no arbitrage conditions (2) are favored by penalization, as opposed to imposed to hold exactly in the previous hard constraint approach.…”

“…Let the function being approximated, p ∈ H α,R ([0, 1] i ) be Hölder smooth with parameters α > 0 and R > 0, where H α,R (Ω) := {p ∈ H α (Ω) : ||p|| H α (Ω) ≤ R}. Then Theorem 4.1 in Ohn and Kim (2019) states the existence of positive constants L 0 , N 0 , Σ 0 , B 0 depending only on i, α, R and ς s.t. for any > 0, the neural network Figure A1 shows the upper bound Σ on the network sparsity, |θ| 0 ≤ Σ, as a function of the error tolerance and Hölder smoothness, α, of the function being approximated.…”

“…We recall a result from Ohn and Kim (2019) which describes how the sparsity in a neural network affects its approximation error.…”

Parametric Flatten-T Swish: An Adaptive Nonlinear Activation Function for Deep Learning

“…The previous works have demonstrated that the locally nonlinear activation functions consistently outperformed multilinear activation functions across the different network architectures on visual recognition tasks. This is proven by a recent theoretical study of Ohn and Kim (2019) that the locally non-linear region can promote better expressivity and non-linear approximation capability.…”

ReLU Network with Bounded Width Is a Universal Approximator in View of an Approximate Identity

“…Actually, the Rectified Linear Units (ReLU) activation function is the most popular choice in practical use of the neural network [12]. In this reason, most of the recent results on the universal approximation theory is about the ReLU network [5,[13][14][15][16][17][18][19][20]. Cohen et al [13] provided the deep convolutional neural network with the ReLU activation function that cannot be realized by a shallow network if the number of nodes of its hidden layer is no more than an exponential bound.…”