On the Spectral Property of Kernel-Based Sensor Fusion Algorithms of High Dimensional Data

Abstract

We study the behavior of two kernel based sensor fusion algorithms, nonparametric canonical correlation analysis (NCCA) and alternating diffusion (AD), under the nonnull setting that the clean datasets collected from two sensors are modeled by a common low dimensional manifold embedded in a high dimensional Euclidean space and the datasets are corrupted by high dimensional noise. We establish the asymptotic limits and convergence rates for the eigenvalues of the associated kernel matrices assuming that the sample dimension and sample size are comparably large, where NCCA and AD are conducted using the Gaussian kernel. It turns out that both the asymptotic limits and convergence rates depend on the signal-to-noise ratio (SNR) of each sensor and selected bandwidths. On one hand, we show that if NCCA and AD are directly applied to the noisy point clouds without any sanity check, it may generate artificial information that misleads scientists' interpretation. On the other hand, we prove that if the bandwidths are selected adequately, both NCCA and AD can be made robust to high dimensional noise when the SNRs are relatively large.

Learning Low-Dimensional Nonlinear Structures from High-Dimensional Noisy Data: An Integral Operator Approach

“…Albeit these endeavors, much less is known when the data is high-dimensional and noisy as modeled by (1.1) and (1.2), nor does the behavior of the associated kernel random matrix K n . In the null case (i.e., y i = z i ), it has been shown in [11,18,21,22,26,29] that when h n = p, the random kernel matrix K n can be well approximated by a low-rank perturbed random Gram matrix. Consequently, studying K n under pure noise is closely related to PCA with some low-rank perturbations.…”

“…Consequently, studying K n under pure noise is closely related to PCA with some low-rank perturbations. Moreover, since in this case the degree matrix is close to a scalar matrix [21], the graph-based methods and the RKHS-based methods are asymptotically equivalent. As for the non-null cases, spectral convergence of K n has been studied in some special cases in [25] when h n = p, and more recently under a general setting in [20].…”

“…There are fundamental advantages of the proposed bandwidth. Under the high-dimensional setup (1.2), most of the existing literature regarding the kernel random matrix K n focus on a pre-specified fixed bandwidth [18,21,22,26]. In particular, when σ = 1, as they mainly deal with the null cases, h n = Cp is chosen for some constant C > 0, since the total noise is tr(Cov(z i )) = p. In contrast, under the non-null regime α > β + 1 considered in the current study, if we still choose h n = Cp, the bandwidth may be too small compared to the signal ( n α ) so that the decay of the kernel function forces each sample to be "blind" of the other nearby samples.…”

“…The global scaling parameter θ characterizes the overall signal-to-noise ratio, which plays the similar role as the term n α−β in our theoretical results. In addition to the proposed adaptive bandwidth selection scheme, we also consider (i) the fixed bandwidth h n ≡ p, which has been used in many existing works such as [18,21,22,25,26,27], and (ii) the data-driven bandwidth selection method, denoted as "SBY," proposed by [58]. We evaluate the above bandwidths by comparing the corresponding low-dimensional spectral embeddings U Ω Λ Ω for Ω = {1, 2, ..., r} of the high-dimensional noisy data {y i } 1≤i≤n , and the actual low-dimensional noiseless samples {x i } 1≤i≤n .…”

“…Many of these results can be found or generalized from [20,21]. We will consistently use stochastic domination to characterize the high-dimensional concentration.…”