Depth Selection for Deep ReLU Nets in Feature Extraction and Generalization

Han, Zhi; Yu, Siquan; Lin, Shao-Bo

doi:10.1109/tpami.2020.3032422

Cited by 27 publications

(31 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…But their results are slightly worse than Theorem 2.1 in this paper, in the sense that there is either an additional logarithmic term or under the weaker norm. Recently, Han et al [17] indicated that deep ReLU nets can achieve the optimal generalization performance for numerous learning tasks, but the depth of [31] is much larger than ours. Recently, Zhou [35,36] also verified that deep convolutional neural network (DCNN) is universal, i.e., DCNN can be used to approximate any continuous function to an arbitrary accuracy when the depth of the neural network is large enough.…”

Section: Related Workmentioning

confidence: 62%

“…All the above literature [34,31,17,35,36] demonstrated that deep nets with ReLU activation function and DCNN have good properties both in approximation and generalization. However, there are too deep to be particularly used in real tasks.…”

Section: Related Workmentioning

confidence: 99%

“…Recent literature in deep nets [34,17] proved that deep nets with ReLU activation function (denoting deep ReLU nets) are more efficiently in approximating smooth function and possess better generalization performance for numerous learning tasks than shallow nets. Nevertheless, the constructed deep ReLU nets are too deep, which results in several difficulty in training, including the gradient vanishing phenomenon and disvergence issue [12].…”

Section: Introductionmentioning

confidence: 99%

“…To this end, we construct in this paper a deep net with three hidden layers that possesses the following properties: localized approximation, optimal approximation rate, controllable parameters, non-saturation and spatial sparsity. Our main tool for analysis is the localized approximation [7,24], "product gate" strategy [9,31,34] and localized Taylor polynomials [17,31].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Approximation smooth and sparse functions by deep neural networks without saturation

Liu¹

2020

Preprint

View full text Add to dashboard Cite

Constructing neural networks for function approximation is a classical and longstanding topic in approximation theory. In this paper, we aim at constructing deep neural networks (deep nets for short) with three hidden layers to approximate smooth and sparse functions. In particular, we prove that the constructed deep nets can reach the optimal approximation rate in approximating both smooth and sparse functions with controllable magnitude of free parameters. Since the saturation that describes the bottleneck of approximate is an insurmountable problem of constructive neural networks, we also prove that deepening the neural network with only one more hidden layer can avoid the saturation. The obtained results underlie advantages of deep nets and provide theoretical explanations for deep learning.

show abstract

Section: Related Workmentioning

confidence: 62%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Approximation smooth and sparse functions by deep neural networks without saturation

Liu¹

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…While it is known that a relation does exist between the network depth and the size of the convolutional filters (and consequently the receptive field) [26], the question of the necessity of depth has not been investigated much. In [27], the authors proposed quantification of the correspondence between features learnt by the network and its depth. DnCNN [12] has been designed following this approach.…”

Section: Introductionmentioning

confidence: 99%

Towards Reduced CNNs for De-Noising Phase Images Corrupted with Speckle Noise

2021

View full text Add to dashboard Cite

Digital holography is a very efficient technique for 3D imaging and the characterization of changes at the surfaces of objects. However, during the process of holographic interferometry, the reconstructed phase images suffer from speckle noise. In this paper, de-noising is addressed with phase images corrupted with speckle noise. To do so, DnCNN residual networks with different depths were built and trained with various holographic noisy phase data. The possibility of using a network pre-trained on natural images with Gaussian noise is also investigated. All models are evaluated in terms of phase error with HOLODEEP benchmark data and with three unseen images corresponding to different experimental conditions. The best results are obtained using a network with only four convolutional blocks and trained with a wide range of noisy phase patterns.

show abstract

Deep Learning + Complex Physics Field Modeling: Illustrated by the Example of Numerical Investigation on Low Temperature Plasma

Zhao,

Pan,

et al. 2023

Lecture Notes in Electrical Engineering

View full text Add to dashboard Cite

Depth Selection for Deep ReLU Nets in Feature Extraction and Generalization

Cited by 27 publications

References 44 publications

Approximation smooth and sparse functions by deep neural networks without saturation

Approximation smooth and sparse functions by deep neural networks without saturation

Towards Reduced CNNs for De-Noising Phase Images Corrupted with Speckle Noise

Deep Learning + Complex Physics Field Modeling: Illustrated by the Example of Numerical Investigation on Low Temperature Plasma

Contact Info

Product

Resources

About