Topological Properties of the Set of Functions Generated by Neural Networks of Fixed Size

Abstract

We analyze the topological properties of the set of functions that can be implemented by neural networks of a fixed size. Surprisingly, this set has many undesirable properties. It is highly non-convex, except possibly for a few exotic activation functions. Moreover, the set is not closed with respect to $$L^p$$ L p -norms, $$0< p < \infty $$ 0 < p < ∞ , for all practically used activation functions, and also not closed with respect to the $$L^\infty $$ L ∞ -norm for all practically used activation functions except for the ReLU and the parametric ReLU. Finally, the function that maps a family of weights to the function computed by the associated network is not inverse stable for every practically used activation function. In other words, if $$f_1, f_2$$ f 1 , f 2 are two functions realized by neural networks and if $$f_1, f_2$$ f 1 , f 2 are close in the sense that $$\Vert f_1 - f_2\Vert _{L^\infty } \le \varepsilon $$ ‖ f 1 - f 2 ‖ L ∞ ≤ ε for $$\varepsilon > 0$$ ε > 0 , it is, regardless of the size of $$\varepsilon $$ ε , usually not possible to find weights $$w_1, w_2$$ w 1 , w 2 close together such that each $$f_i$$ f i is realized by a neural network with weights $$w_i$$ w i . Overall, our findings identify potential causes for issues in the training procedure of deep learning such as no guaranteed convergence, explosion of parameters, and slow convergence.

Invariant spectral foliations with applications to model order reduction and synthesis

“…MAP means a polynomial model fit was carried out first, DATA means that the ISF was directly fitted to the data, O(α) indicates that order α polynomials were used deep neural networks [5,12], or other kinds of nonlinear approximation methods [9], that allow to represent high-dimensional functions with reasonable efficiency as opposed to polynomials. The challenge with nonlinear approximations, in particular with neural networks, is that they can be difficult to fit to data, because the dis-tance between parameters that provide small improvements in accuracy can be large and therefore not easy to find [21]. Nevertheless, deep neural networks have enabled great advances in many fields of engineering and therefore this approach will be explored elsewhere.…”

“…We can accurately identify the dynamics on an ISF and determine its instantaneous damping ratio (22) and angular frequency (21). It is, however, not possible to attach a unique amplitude to a leaf within a foliation.…”

Best k-Layer Neural Network Approximations

“…We would like to emphasize that the finite-dimensional problem (5) is not any easier than the infinite-dimensional problem (4); they simply require different techniques. In particular, our results do not follow from the results in [3,7,14] for infinitedimensional spaces -we will have more to say about this in Section 2.…”

The Universal Approximation Property

“…The construction of Example 4 parallels the formulation given in [46,47]. However, in [47] elements of F are referred to as neural networks and functions in N N (F , ) are called their realizations.…”