Fast Gaussian Process Regression for Big Data

Abstract

Gaussian Processes are widely used for regression tasks. A known limitation in the application of Gaussian Processes to regression tasks is that the computation of the solution requires performing a matrix inversion. The solution also requires the storage of a large matrix in memory. These factors restrict the application of Gaussian Process regression to small and moderate size data sets. We present an algorithm that combines estimates from models developed using subsets of the data obtained in a manner similar to the bootstrap. The sample size is a critical parameter for this algorithm. Guidelines for reasonable choices of algorithm parameters, based on detailed experimental study, are provided. Various techniques have been proposed to scale Gaussian Processes to large scale regression tasks. The most appropriate choice depends on the problem context. The proposed method is most appropriate for problems where an additive model works well and the response depends on a small number of features. The minimax rate of convergence for such problems is attractive and we can build effective models with a small subset of the data. The Stochastic Variational Gaussian Process and the Sparse Gaussian Process are also appropriate choices for such problems. These methods pick a subset of data based on theoretical considerations. The proposed algorithm uses bagging and random sampling. Results from experiments conducted as part of this study indicate that the algorithm presented in this work can be as effective as these methods.

A Bayesian perspective of statistical machine learning for big data

“…Hypothesis classes other than trees can be used with these techniques. For example, Das, Roy and Sambasivan (2018) apply bagging using Gaussian Processes prior regression on big datasets. Model development in ensemble methods, such as xgboost, is performed on a representative sample from the data, see e.g., Chen and Guestrin (2016).…”

“…Das, Roy and Sambasivan (2018) illustrate an application of the ensemble technique to a dataset from the utility domain that has nearly two million data instances, see Dheeru and Karra Taniskidou (2017) for further detail of the dataset. The dataset represents electricity consumption data at oneminute sampling intervals from a household.…”

“…The learning task is to predict the voltage from the other features. Das, Roy and Sambasivan (2018) use bootstrapped samples to develop a Gaussian Process model for the regression task. This is very similar to the use of a random sample to develop a tree model in Breiman (2001).…”

“…This is very similar to the use of a random sample to develop a tree model in Breiman (2001). However, a Gaussian Process model is developed using the sample in Das, Roy and Sambasivan (2018). Kernel selection for the Gaussian Process regression model is determined by using data exploration and experimental evaluation.…”

“…As discussed in Das, Roy and Sambasivan (2018), in many problems it is observed that data usually lie in a manifold Yang et al (2015) or depend on a small number of features. In such problems it is possible to develop reasonable models with a relatively small sample (see Das, Roy and Sambasivan (2018)[Section 3] for details). The sample size is a critical parameter of the algorithm used in Das, Roy and Sambasivan (2018).…”