Variational Gaussian Processes: A Functional Analysis View

Abstract

Variational Gaussian process (GP) approximations have become a standard tool in fast GP inference. This technique requires a user to select variational features to increase efficiency. So far the common choices in the literature are disparate and lacking generality. We propose to view the GP as lying in a Banach space which then facilitates a unified perspective. This is used to understand the relationship between existing features and to draw a connection between kernel ridge regression and variational GP approximations.

Generalized Variational Inference in Function Spaces: Gaussian Measures meet Bayesian Deep Learning

“…A true Bayesian is committed to the use of the KL divergence in (2) as maximizing L is equivalent to minimizing the KL divergence between the true posterior measure and the variational measure. This equivalence is typically demonstrated using pdfs but the argument generalizes to infinite dimensions as is shown for GPs in Matthews et al [2016] or in a more measure theoretic formulation in Theorem 4 of Wild and Wynne [2021].…”

“…It shall be noted that there is no need to appeal to GPs in order to justify the use of GMs. In fact, it has recently been demonstrated that variational inference for GPs can be formulated purely in terms of GMs [Wild and Wynne, 2021]. In the following sections we will therefore deploy GMs without any reference to GPs, but it is of course always possible to think of them as the measures that correspond to GPs where the kernel satisfies an additional assumption such as (8).…”