Predictive Approaches for Choosing Hyperparameters in Gaussian Processes

Abstract

Gaussian processes are powerful regression models specified by parameterized mean and covariance functions. Standard approaches to choose these parameters (known by the name hyperparameters) are maximum likelihood and maximum a posteriori. In this article, we propose and investigate predictive approaches based on Geisser's predictive sample reuse (PSR) methodology and the related Stone's cross-validation (CV) methodology. More specifically, we derive results for Geisser's surrogate predictive probability (GPP), Geisser's predictive mean square error (GPE), and the standard CV error and make a comparative study. Within an approximation we arrive at the generalized cross-validation (GCV) and establish its relationship with the GPP and GPE approaches. These approaches are tested on a number of problems. Experimental results show that these approaches are strongly competitive with the existing approaches.

View-Constrained Latent Variable Model for Multi-view Facial Expression Classification

“…On the other hand, the problem in Eq. (14) is similar to that of the regularized Kernel Ridge Regression (KRR) [21] and has a closed-form solution:…”

“…To alleviate this, we use the notion of the Leave-One-Out (LOO) cross-validation procedure for the KRR [21] to define learning of γ (v) and λ (v) , and, thus, obtain A (v) indirectly. The learning in LOO is based on the fact that given any training set and the corresponding regression model, if we add a sample to the training set with the target equal to the output predicted by the model, the latter will not change since the cost will not increase [21]. Thus, given the training set with the sample y …”

Learning Non-stationary System Dynamics Online Using Gaussian Processes

“…In our current system, we employ leave-one-out cross-validation to adapt the hyper-parameters. Alternative cross-validation approaches in the context of GPs are described by Sundararajan and Keerthi [10]. In general, one seeks to find hyper-parameters that minimize the average loss of all training samples given a predefined optimization criterion.…”

Learning Depth from Stereo

“…An alternative way of inferring θ is to use a Bayesian variant of the leave-oneout error (GPP, Geisser's surrogate predictive probability, [4]). In our study we will use both methods, choosing the most appropriate one for each of our two covariance functions.…”

“…Indeed, the lengthscales λ d can grow to eliminate the contribution of any irrelevant input dimension. The parameters σ 2 ν , σ 2 and g of the polynomial covariance function were estimated by maximizing the GPP criterion [4]. The parameters σ 2 ν , σ 2 and the λ d of the squared exponential kernel were estimated by maximizing their marginal log likelihood [5].…”