Function Approximation by Deep Neural Networks with Parameters $$\{0,\pm \frac{1}{2}, \pm 1, 2\}$$

Abstract

In this paper, it is shown that $$C_\beta $$ C β -smooth functions can be approximated by deep neural networks with ReLU activation function and with parameters $$\{0,\pm \frac{1}{2}, \pm 1, 2\}$$ { 0 , ± 1 2 , ± 1 , 2 } . The $$l_0$$ l 0 and $$l_1$$ l 1 parameter norms of considered networks are thus equivalent. The depth, the width and the number of active parameters of the constructed networks have, up to a logarithmic factor, the same dependence on the approximation error as the networks with parameters in $$[-1,1]$$ [ - 1 , 1 ] . In particular, this implies that the nonparametric regression estimation with constructed networks achieves, up to logarithmic factors, the same minimax convergence rates as with sparse networks with parameters in $$[-1,1]$$ [ - 1 , 1 ] .

Expressive power of binary and ternary neural networks

“…Approximations of β-Hölder functions belonging to the ball [9]. Using that result and substituting the interval [−1, 1] by the finite set of weights {0, ± 1 2 , ±1, 2}, the following theorem is given in [2]:…”

Nonparametric regression with modified ReLU networks

“…(i) deriving prediction rates of the empirical risk minimizers (1) or ( 2); (ii) finding an optimization algorithm that identifies the corresponding empirical risk minimizers. Convergence rates of empirical risk minimizers (ERM) over the classes of deep ReLU networks are studied in [4], [13], [15] and [18]. In [4] it is shown that the ERM of the form (1), with W n being the set of weight vectors with coordinates {0, ±1/2, ±1, 2}, attains, up to logarithmic factors, the minimax rates of prediction of β-smooth functions.…”

“…Convergence rates of empirical risk minimizers (ERM) over the classes of deep ReLU networks are studied in [4], [13], [15] and [18]. In [4] it is shown that the ERM of the form (1), with W n being the set of weight vectors with coordinates {0, ±1/2, ±1, 2}, attains, up to logarithmic factors, the minimax rates of prediction of β-smooth functions. The finiteness of W n guarantees that in this case the ERM can be found within finitely many steps.…”

Neural networks with superexpressive activations and integer weights

“…We will consider the family of activation functions A = {σ, ⌊•⌋}, where the role of the activation σ is to guarantee that the conditions of the Kronecker's theorem are satisfied and that it gives a small range for the integer multipliers. Having this range we then bound the entropy of approximant networks and use this bound to get for β-Hölder continuous regression functions a convergence rate of order n −2β 2β+d log 2 n. Note that our approach is in some sense opposite to the one given in [1], where approximations by deep networks with weights {0, ± 1 2 , ±1, 2} are considered: in one case we fix a finite set of weights and adjust the network architecture and in the other case we fix the network architecture and adjust the integer weights to attain a certain approximation rate.…”