Nonlinear Approximation and (Deep) $$\mathrm {ReLU}$$ Networks

Daubechies, Ingrid; DeVore, Ronald A.; Foucart, Simon; Hanin, Boris; Petrova, Guergana

doi:10.1007/s00365-021-09548-z

Cited by 127 publications

(69 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the second step, we construct a ReLU neural network with desired size N and that realizes φ exactly. This construction is based on results from [12] for representing free-knot linear splines and sums of functions using neural networks. The improvement over [11] results from this second step.…”

Section: The Achievability Part: Proof Of Theoremmentioning

confidence: 99%

“…In other regimes, our construction uses at most the same number of neurons as that of [11]. The key ingredient that we used to reduce the number of neurons is an efficient construction of the neural networks that compute piecewise affine functions studied in [12]. Instead of representing a piecewise affine function by a straightforward neural net of depth 2 and size linear in the number of affine pieces, [12] constructs a deeper neural network computing the same function.…”

Section: Introductionmentioning

confidence: 99%

“…The key ingredient that we used to reduce the number of neurons is an efficient construction of the neural networks that compute piecewise affine functions studied in [12]. Instead of representing a piecewise affine function by a straightforward neural net of depth 2 and size linear in the number of affine pieces, [12] constructs a deeper neural network computing the same function. Suitably choosing the depth of this neural net, we are able to reduce the size of the network compared to the straightforward network.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Approximating Probability Distributions by ReLU Networks

Mukherjee¹,

Tchamkerten²,

Yousefi³

2021

Preprint

View full text Add to dashboard Cite

How many neurons are needed to approximate a target probability distribution using a neural network with a given input distribution and approximation error? This paper examines this question for the case when the input distribution is uniform, and the target distribution belongs to the class of histogram distributions. We obtain a new upper bound on the number of required neurons, which is strictly better than previously existing upper bounds. The key ingredient in this improvement is an efficient construction of the neural nets representing piecewise linear functions. We also obtain a lower bound on the minimum number of neurons needed to approximate the histogram distributions.

show abstract

Section: The Achievability Part: Proof Of Theoremmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Approximating Probability Distributions by ReLU Networks

Mukherjee¹,

Tchamkerten²,

Yousefi³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…A classical result along those lines is the universal approximation theorem [7,13], which states that singlehidden-layer neural networks with sigmoidal activation function can approximate continuous functions on compact subsets of R n arbitrarily well. More recent developments in this area are concerned with the influence of network depth on attainable approximation quality [8,10,23]. A theory establishing the fundamental limits of deep neural network expressivity is provided in [6,9].…”

Section: Introductionmentioning

confidence: 99%

High-dimensional distribution generation through deep neural networks

Perekrestenko¹,

Eberhard

Bölcskei

2021

Partial Differ. Equ. Appl.

View full text Add to dashboard Cite

We show that every d-dimensional probability distribution of bounded support can be generated through deep ReLU networks out of a 1-dimensional uniform input distribution. What is more, this is possible without incurring a cost—in terms of approximation error measured in Wasserstein-distance—relative to generating the d-dimensional target distribution from d independent random variables. This is enabled by a vast generalization of the space-filling approach discovered in Bailey and Telgarsky (in: Bengio (eds) Advances in neural information processing systems vol 31, pp 6489–6499. Curran Associates, Inc., Red Hook, 2018). The construction we propose elicits the importance of network depth in driving the Wasserstein distance between the target distribution and its neural network approximation to zero. Finally, we find that, for histogram target distributions, the number of bits needed to encode the corresponding generative network equals the fundamental limit for encoding probability distributions as dictated by quantization theory.

show abstract

“…A classical result along those lines is the universal approximation theorem [7], [13], which states that single-hidden-layer neural networks with sigmoidal activation function can approximate continuous functions on compact subsets of R n arbitrarily well. More recent developments in this area are concerned with the influence of network depth on attainable approximation quality [23], [8], [10]. A theory establishing the fundamental limits of deep neural network expressivity is provided in [6], [9].…”

Section: Introductionmentioning

confidence: 99%

High-Dimensional Distribution Generation Through Deep Neural Networks

Perekrestenko¹,

Eberhard²,

Bölcskei³

2021

Preprint

View full text Add to dashboard Cite

We show that every d-dimensional probability distribution of bounded support can be generated through deep ReLU networks out of a 1-dimensional uniform input distribution. What is more, this is possible without incurring a cost-in terms of approximation error measured in Wasserstein-distance-relative to generating the d-dimensional target distribution from d independent random variables. This is enabled by a vast generalization of the space-filling approach discovered in [2]. The construction we propose elicits the importance of network depth in driving the Wasserstein distance between the target distribution and its neural network approximation to zero. Finally, we find that, for histogram target distributions, the number of bits needed to encode the corresponding generative network equals the fundamental limit for encoding probability distributions as dictated by quantization theory.

show abstract

Nonlinear Approximation and (Deep) $$\mathrm {ReLU}$$ Networks

Cited by 127 publications

References 22 publications

Approximating Probability Distributions by ReLU Networks

Approximating Probability Distributions by ReLU Networks

High-dimensional distribution generation through deep neural networks

High-Dimensional Distribution Generation Through Deep Neural Networks

Contact Info

Product

Resources

About