Connections and Equivalences between the Nyström Method and Sparse Variational Gaussian Processes

Abstract

We investigate the connections between sparse approximation methods for making kernel methods and Gaussian processes (GPs) scalable to massive data, focusing on the Nyström method and the Sparse Variational Gaussian Processes (SVGP). While sparse approximation methods for GPs and kernel methods share some algebraic similarities, the literature lacks a deep understanding of how and why they are related. This is a possible obstacle for the communications between the GP and kernel communities, making it difficult to transfer results from one side to the other. Our motivation is to remove this possible obstacle, by clarifying the connections between the sparse approximations for GPs and kernel methods. In this work, we study the two popular approaches, the Nyström and SVGP approximations, in the context of a regression problem, and establish various connections and equivalences between them. In particular, we provide an RKHS interpretation of the SVGP approximation, and show that the Evidence Lower Bound of the SVGP contains the objective function of the Nyström approximation, revealing the origin of the algebraic equivalence between the two approaches. We also study recently established convergence results for the SVGP and how they are related to the approximation quality of the Nyström method.

Variational Gaussian Processes: A Functional Analysis View

“…The GRE view is essential to the proof. The result generalises the connection between Nyström KRR and inducing points outlined in Wild et al (2021) and opens the door to applying the theory of Nyström KRR error bounds to variational GPs to gain a better understanding of the latter's approximation properties. Vice versa, recent advances in variational GPR approaches, for example, variational Fourier features, could be leveraged in the context of KRR Nyström, as they simply correspond to a particular choice of M.…”

“…In this section we demonstrate how the GRE framework reveals further connections between Gaussian process regression and the Nyström approximation for kernel Ridge regression (KRR). These connections have been known for a while and received some attention recently (Parzen, 1961;Wahba, 1990;Kanagawa et al, 2018;Wild et al, 2021) Let k be a kernel, H k the corresponding RKHS and {x n , y n } N n=1 ⊂ X × R be paired observations. In Nyström KRR (Williams and Seeger, 2001) we seek to minimise the empirical risk over a finite dimensional subspace…”

Improved Convergence Rates for Sparse Approximation Methods in Kernel-Based Learning

“…Let Z m = [z 1 , z 2 , ..., z m ] ⊤ ∈ X m be the set of inducing points. The approximate regressor and posterior covariance of the (surrogate) GP model are given as [Wild et al, 2021]:…”

“…In the KRR approach, μn is interpreted as the regularized least squares estimator within a reduced rank RKHS induced by Z m [e.g., see, Wild et al, 2021]. In particular, define…”