An analysis of training and generalization errors in shallow and deep networks

Abstract

This paper is motivated by an open problem around deep networks, namely, the apparent absence of overfitting despite large over-parametrization which allows perfect fitting of the training data. In this paper, we analyze this phenomenon in the case of regression problems when each unit evaluates a periodic activation function. We argue that the minimal expected value of the square loss is inappropriate to measure the generalization error in approximation of compositional functions in order to take full advantage of the compositional structure. Instead, we measure the generalization error in the sense of maximum loss, and sometimes, as a pointwise error. We give estimates on exactly how many parameters ensure both zero training error as well as a good generalization error. We prove that a solution of a regularization problem is guaranteed to yield a good training error as well as a good generalization error and estimate how much error to expect at which test data.

A manifold learning approach for gesture recognition from micro-Doppler radar measurements

“…In this range the Gaussian kernel is minimal at r = 3 and r = 4 for the the binary and normalized versions, respectively; then as the dimension increases so does the classification accuracy, reaching 50% by r = 15 and 60% for q = 20. Interestingly, the performance of the Grassman kernel stays constant for r ∈ [1,20] with the normalized version maintaining an error of 15% and the binary at 18%. For r > 9 the Laplacian kernel obtained roughly the same performance, with the binary feature out performing by 3%, and the Laplacian kernel outperformed the Grassman kernel at larger values of r. We attribute this drop-off in accuracy for increasing r, due to the fact the as dimension of the manifold increases the data points become more spread apart (less dense) and a poorer approximation of the labeling function is achieved.…”

“…In particular, it is shown in [23] that substituting the ReLU activation function by a polynomial approximation exhibits the same behavior as the original network. In [20] we have analyzed the question from the point of view of approximation theory so as to examine the intrinsic features of the data (rather than focusing on specific training algorithms) that allow this phenomenon. A crucial role in the proofs of the results in that paper is played by a highly localized kernel.…”

“…A crucial role in the proofs of the results in that paper is played by a highly localized kernel. In this paper, we focus on the properties of a localized kernel in a much greater setting of a locally compact metric measure space that allow the results analogous to those in [20].…”

“…20) that|P T (τ ; V(C); F )(x)| ≤ k:y k ∈∆(x,η/3) |a * k ||Φ N (x, y k )| N −q σ N (F f 0 ) ∞ y∈∆(x,η/3)…”

Dimension independent bounds for general shallow networks

“…• Our bounds are in the uniform norm. We have argued in [25] that the usual measurement of generalization error using the expected value of the least square loss is not applicable for approximation theory for deep networks; one has to use the uniform approximation to take full advantage of the compositional structure.…”