Analysis of Some Methods for Reduced Rank Gaussian Process Regression

Abstract

While there is strong motivation for using Gaussian Processes (GPs) due to their excellent performance in regression and classification problems, their computational complexity makes them impractical when the size of the training set exceeds a few thousand cases. This has motivated the recent proliferation of a number of cost-effective approximations to GPs, both for classification and for regression. In this paper we analyze one popular approximation to GPs for regression: the reduced rank approximation. While generally GPs are equivalent to infinite linear models, we show that Reduced Rank Gaussian Processes (RRGPs) are equivalent to finite sparse linear models. We also introduce the concept of degenerate GPs and show that they correspond to inappropriate priors. We show how to modify the RRGP to prevent it from being degenerate at test time. Training RRGPs consists both in learning the covariance function hyperparameters and the support set. We propose a method for learning hyperparameters for a given support set. We also review the Sparse Greedy GP (SGGP) approximation (Smola and Bartlett, 2001), which is a way of learning the support set for given hyperparameters based on approximating the posterior. We propose an alternative method to the SGGP that has better generalization capabilities. Finally we make experiments to compare the different ways of training a RRGP. We provide some Matlab code for learning RRGPs

Sequential Model-Based Parameter Optimization: an Experimental Investigation of Automated and Interactive Approaches

“…Inverting the n × n matrix [K K K + σ 2 I] takes time O(n 3 ), and has to be done in every step of the hyper-parameter optimization. Various approximations can be used to reduce this to O(n 2 ); see, e.g., Quinonero-Candela et al (2007). We refer to the process of optimizing hyper-parameters and computing the inverse as fitting the model.…”

An efficient algorithm for learning to rank from preference graphs

“…In this section, we propose an approach that is based on a similar kind of idea as the subset of regressors method (see e.g. Poggio and Girosi 1990;Smola and Schölkopf 2000;Rifkin et al 2003;Quiñonero-Candela et al 2007) for the standard regularized least-squares regression. More detailed considerations and experimental results of this approach for RankRLS are presented in Tsivtsivadze et al (2008).…”

“…RLS-based learning algorithms can also be extended for large-scale learning using the subset of regressors method (see e.g. Quiñonero-Candela et al 2007;Tsivtsivadze et al 2008). Further advantages include the possibility to learn several functions in parallel as considered by Rifkin and Klautau (2004).…”

Indoor positioning using multi-frequency RSS with foot-mounted INS

“…The main bottleneck is the inversion of the covariance matrix which has computational complexity O(n 3 ), where n is the number of data points used to construct the GP. One popular solution is to use a sparse approximation of the full GP and a number of available methods are compared and analyzed in [28]. This paper picks up on the ideas used in [29] where an assumption is made about the structure of the kernel function instead of the conditional distribution or the likelihood.…”