Limitations of shallow nets approximation

Lin, Shao-Bo

doi:10.1016/j.neunet.2017.06.016

Cited by 31 publications

(24 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Limitations of the approximation capabilities of shallow nets were firstly proposed in [4] in terms of their incapability of localized approximation. Five years later, [8] described their limitations via providing lower bounds of approximation of smooth functions in the minimax sense, which was recently highlighted by [25] via showing that there exists a probabilistic measure, under which, all smooth functions cannot be approximated by shallow nets very well with high confidence. In [1], Bengio et al also pointed out the limitations of some shallow nets in terms of the so-called "curse of dimensionality".…”

Section: B Learning Rate Analysismentioning

confidence: 99%

Generalization and Expressivity for Deep Nets

Lin

2019

IEEE Trans. Neural Netw. Learning Syst.

Self Cite

View full text Add to dashboard Cite

Along with the rapid development of deep learning in practice, theoretical explanations for its success become urgent. Generalization and expressivity are two widely used measurements to quantify theoretical behaviors of deep learning. The expressivity focuses on finding functions expressible by deep nets but cannot be approximated by shallow nets with the similar number of neurons. It usually implies the large capacity. The generalization aims at deriving fast learning rate for deep nets. It usually requires small capacity to reduce the variance. Different from previous studies on deep learning, pursuing either expressivity or generalization, we take both factors into account to explore theoretical advantages of deep nets. For this purpose, we construct a deep net with two hidden layers possessing excellent expressivity in terms of localized and sparse approximation. Then, utilizing the well known covering number to measure the capacity, we find that deep nets possess excellent expressive power (measured by localized and sparse approximation) without essentially enlarging the capacity of shallow nets. As a consequence, we derive near optimal learning rates for implementing empirical risk minimization (ERM) on the constructed deep nets. These results theoretically exhibit the advantage of deep nets from learning theory viewpoints.

show abstract

Section: B Learning Rate Analysismentioning

confidence: 99%

Generalization and Expressivity for Deep Nets

Lin

2019

IEEE Trans. Neural Netw. Learning Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

“…We refer the reader to [5,25] for interesting results and discussions around other limitations of such networks.…”

Section: Analysis Of the Multivariate Casementioning

confidence: 99%

On the approximation by single hidden layer feedforward neural networks with fixed weights

2018

View full text Add to dashboard Cite

Abstract. Feedforward neural networks have wide applicability in various disciplines of science due to their universal approximation property. Some authors have shown that single hidden layer feedforward neural networks (SLFNs) with fixed weights still possess the universal approximation property provided that approximated functions are univariate. But this phenomenon does not lay any restrictions on the number of neurons in the hidden layer. The more this number, the more the probability of the considered network to give precise results. In this note, we constructively prove that SLFNs with the fixed weight 1 and two neurons in the hidden layer can approximate any continuous function on a compact subset of the real line. The applicability of this result is demonstrated in various numerical examples. Finally, we show that SLFNs with fixed weights cannot approximate all continuous multivariate functions.

show abstract

“…This theorem shows a particular limitation of neural networks with one hidden layer. We refer the reader to [5,25] for interesting results and discussions around other limitations of such networks.…”

Section: Analysis Of the Multivariate Casementioning

confidence: 99%

On the approximation by single hidden layer feedforward neural networks with fixed weights

Guliyev¹,

Ismailov²

2017

Preprint

View full text Add to dashboard Cite

Feedforward neural networks have wide applicability in various disciplines of science due to their universal approximation property. Some authors have shown that single hidden layer feedforward neural networks (SLFNs) with fixed weights still possess the universal approximation property provided that approximated functions are univariate. But this phenomenon does not lay any restrictions on the number of neurons in the hidden layer. The more this number, the more the probability of the considered network to give precise results. In this note, we constructively prove that SLFNs with the fixed weight 1 and two neurons in the hidden layer can approximate any continuous function on a compact subset of the real line. The applicability of this result is demonstrated in various numerical examples. Finally, we show that SLFNs with fixed weights cannot approximate all continuous multivariate functions.2010 Mathematics Subject Classification. 41A30, 41A63, 65D15, 68T05, 92B20.

show abstract

Limitations of shallow nets approximation

Cited by 31 publications

References 26 publications

Generalization and Expressivity for Deep Nets

Generalization and Expressivity for Deep Nets

On the approximation by single hidden layer feedforward neural networks with fixed weights

On the approximation by single hidden layer feedforward neural networks with fixed weights

Contact Info

Product

Resources

About