Compression, Inversion, and Approximate PCA of Dense Kernel Matrices at Near-Linear Computational Complexity

Abstract

Dense kernel matrices Θ ∈ R N ×N obtained from point evaluations of a covariance function G at locations {x i } 1≤i≤N arise in statistics, machine learning, and numerical analysis. For covariance functions that are Green's functions elliptic boundary value problems and approximately equally spaced sampling points, we show how to identify a subset S ⊂ {1, . . . , N } × {1, . . . , N }, with #S = O N log(N ) log d (N/ ) , such that the zero fill-in block-incomplete Cholesky decomposition of Θ i,j 1 (i,j)∈S is an -approximation of Θ. This block-factorisation can provably be obtained in O N log 2 (N ) log (1/ ) + log 2 (N ) 4d+1 complexity in time. Numerical evidence further suggests that element-wise Cholesky decomposition with the same ordering constitutes an O N log 2 (N ) log 2d (N/ ) solver. The algorithm only needs to know the spatial configuration of the x i and does not require an analytic representation of G. Furthermore, an approximate PCA with optimal rate of convergence in the operator norm can be easily read off from this decomposition. Hence, by using only subsampling and the incomplete Cholesky decomposition, we obtain at nearly linear complexity the compression, inversion and approximate PCA of a large class of covariance matrices. By inverting the order of the Cholesky decomposition we also obtain a near-linear-time solver for elliptic PDEs.

GParareal: A time-parallel ODE solver using Gaussian process emulation

“…First, an increasing number of data points, O(kJ), is problematic for the standard cubic complexity GP implemented here. In this case, a more sophisticated (non-cubic complexity) emulator or using sparse approximate linear algebra techniques (Schäfer et al, 2021) would be beneficial. Second, trying to emulate a d-dimensional function F − G is difficult if the number of evaluation points is not sufficient.…”

Bayesian Nonstationary and Nonparametric Covariance Estimation for Large Spatial Data (with Discussion)

“…. , s n , and hence the columns of Y, are ordered according to a maximin ordering (Guinness, 2018;Schäfer et al, 2021b), which sequentially selects each location in the ordering to maximize the minimum distance from locations already selected (see Figure 2).…”

“…Our model can be viewed as a nonparametric extension of the Vecchia approach, as regularized inference on a sparse Cholesky factor of the precision matrix, or as a series of Bayesian linear regression or spatial prediction problems. We specify prior distributions that are motivated by recent results (Schäfer et al, 2021b,a) on the exponential decay of the entries of the inverse Cholesky factor for Matérn-type covariances under a maximum-minimum-distance ordering of the spatial locations (Guinness, 2018;Schäfer et al, 2021b). Thus, we obtain a highly flexible method that enforces neither stationary nor parametric covariance structures, but instead regularizes the estimation and accounts for uncertainty via Bayesian priors.…”

Bayesian numerical methods for nonlinear partial differential equations

“…For comparison, direct Gaussian conditioning on the information in Lemma 2.1 would incur a higher computational cost of O(n 3 m 3 ), but would provide the joint distribution over the solution u(t, •) at all times t ∈ [0, T ]. Although we do not pursue it in this paper, in the latter case the grid structure present in t and x could be exploited to mitigate the O(n 3 m 3 ) cost; for example, a compactly supported covariance model would reduce the cost by a constant factor (Gneiting 2002), or if the preconditions of Schäfer et al (2021) are satisfied then their approach would reduce the cost to O(nm log(nm) log d+1 (nm/ )) at the expense of introducing an error of O( ). See also the recent work of de Roos et al (2021).…”