The spectrum of kernel random matrices

Abstract

We place ourselves in the setting of high-dimensional statistical inference where the number of variables $p$ in a dataset of interest is of the same order of magnitude as the number of observations $n$. We consider the spectrum of certain kernel random matrices, in particular $n\times n$ matrices whose $(i,j)$th entry is $f(X_i'X_j/p)$ or $f(\Vert X_i-X_j\Vert^2/p)$ where $p$ is the dimension of the data, and $X_i$ are independent data vectors. Here $f$ is assumed to be a locally smooth function. The study is motivated by questions arising in statistics and computer science where these matrices are used to perform, among other things, nonlinear versions of principal component analysis. Surprisingly, we show that in high-dimensions, and for the models we analyze, the problem becomes essentially linear--which is at odds with heuristics sometimes used to justify the usage of these methods. The analysis also highlights certain peculiarities of models widely studied in random matrix theory and raises some questions about their relevance as tools to model high-dimensional data encountered in practice.Comment: Published in at http://dx.doi.org/10.1214/08-AOS648 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

The Spectrum of Random Kernel Matrices: Universality Results for Rough and Varying Kernels

“…Following previous literature [17,13], in this paper these matrices will be refered to as random kernel matrices generated by F and the distribution of X i 's. As described in [17], practical examples of F are of the form…”

“…This research direction has been investigated recently by El Karoui [17] and ChengSinger [13], motivated by studies from machine learning and statistical analysis. In [17], it was assumed that A is generated by either the inner-product or the distance kernels, and with p-independent envelope functions (which is the natural setting relative to the above normalization of X i ). It was shown in [17] that for f sufficiently smooth the limiting behavior of ρ A depends only on a linear component of f .…”

“…We consider random matrices whose entries are f (X T i X j ) or f ( X i − X j 2 ) for iid vectors X i ∈ R p with normalized distribution. Assuming that f is sufficiently smooth and the distribution of X i 's is sufficiently nice, El Karoui [17] showed that the spectral distributions of these matrices behave as if f is linear in the Marčhenko-Pastur limit. When X i 's are Gaussian vectors, variants of this phenomenon were recently proved for varying kernels, i.e.…”

Understanding big data spectral clustering

“…In particular, to the authors' knowledge, there exists no contribution addressing the case of arbitrary p and n. In this article, we propose a new approach consisting in assuming that both p and n are large, and exploiting recent results from random matrix theory. Our method is inspired by [3] which studies the asymptotic distribution of the eigenvalues of K for i.i.d. vectors x i .…”

“…vectors x i . We generalize here [3] by assuming that the x i 's are drawn from a mixture of k Gaussian vectors having means µ 1 , . .…”