Deep Nets for Local Manifold Learning

Abstract

The problem of extending a function f defined on a training data C on an unknown manifold X to the entire manifold and a tubular neighborhood of this manifold is considered in this paper. For X embedded in a high dimensional ambient Euclidean space R D , a deep learning algorithm is developed for finding a local coordinate system for the manifold without eigen-decomposition, which reduces the problem to the classical problem of function approximation on a low dimensional cube. Deep nets (or multilayered neural networks) are proposed to accomplish this approximation scheme by using the training data. Our methods do not involve such optimization techniques as back-propagation, while assuring optimal (a priori) error bounds on the output in terms of the number of derivatives of the target function. In addition, these methods are universal, in that they do not require a prior knowledge of the smoothness of the target function, but adjust the accuracy of approximation locally and automatically, depending only upon the local smoothness of the target function. Our ideas are easily extended to solve both the pre-image problem and the out-of-sample extension problem, with a priori bounds on the growth of the function thus extended.

Construction of Neural Networks for Realization of Localized Deep Learning

“…On the other hand, it was proposed in [1,12] with practical arguments, that deep nets can tackle data in highly-curved manifolds, while any shallow net fails. These arguments were theoretically verified in [9,41], with the implication that adding hidden layers to shallow nets should enable the neural networks to have the capability of processing massive data in a high-dimensional space from samples in lower dimensional manifolds. More precisely, it follows from [13,41] that for a lower d-dimensional connected and compact C ∞ Riemannian submanifold X ⊆ [−1, 1] D (without boundary), isometrically embedded in R D and endowed with the geodesic distance d G , there exists some δ > 0, such that for any x, x ′ ∈ X , with d G (x, x ′ ) < δ,…”

“…In this section, we present a construction of deep neural networks (called deep nets, for simplicity) with three hidden layers to realize certain deep learning algorithms, by applying the mathematical tools of localized approximation in [7], local manifold learning in [9], and local average arguments in [19]. Throughout this paper, we will consider only two activation functions: the Heaviside functionσ 0 and the square-rectifier σ 2 , where the standard notation t + = max{0, t} is used to define σ n (t) = t n + = (t + ) n , for any non-negative integer n.…”

“…In the following, let B G (ξ 0 , τ ), B D (ξ 0 , τ ), and B d (ξ 0 , τ ) denote the closed geodesic ball, the D-dimensional Euclidean ball, and the d-dimensional Euclidean ball, with center at ξ 0 , respectively, and with radius τ > 0. Then the following proposition is a brief summary of Theorem 2.2, Theorem 2.3 and Remark 2.1 in [9], with the implication that neural network can be used as a dimensionality-reduction tool.…”

“…the sample set, and construct a DN with three hidden layers, with the first for the dimensionality-reduction, the second for bias-reduction, and the third for variance-reduction. The main tools for our construction are the "local manifold learning" for deep nets in [9], "localized approximation" for deep nets in [7], and "local average" in [19]. We will also introduce a feedback procedure to eliminate outliers during the learning process.…”

Deep neural networks for rotation-invariance approximation and learning

“…In this era of big data, data-sets of massive size and with various features are routinely acquired, creating a crucial challenge to machine learning in the design of learning strategies for data management, particularly in realization of certain data features. Deep learning [11] is a state-of-the-art approach for the purpose of realizing such features, including localized position information [3,5], geometric structures of data-sets [4,29], and data sparsity [17,18]. For this and other reasons, deep learning has recently received much attention, and has been successful in various application domains [8], such as computer vision, speech recognition, image classification, fingerprint recognition and earthquake forecasting.…”