Understanding and comparing scalable Gaussian process regression for big data

Abstract

As a non-parametric Bayesian model which produces informative predictive distribution, Gaussian process (GP) has been widely used in various fields, like regression, classification and optimization. The cubic complexity of standard GP however leads to poor scalability, which poses challenges in the era of big data. Hence, various scalable GPs have been developed in the literature in order to improve the scalability while retaining desirable prediction accuracy. This paper devotes to investigating the methodological characteristics and performance of representative global and local scalable GPs including sparse approximations and local aggregations from four main perspectives: scalability, capability, controllability and robustness. The numerical experiments on two toy examples and five real-world datasets with up to 250K points offer the following findings. In terms of scalability, most of the scalable GPs own a time complexity that is linear to the training size. In terms of capability, the sparse approximations capture the long-term spatial correlations, the local aggregations capture the local patterns but suffer from over-fitting in some scenarios. In terms of controllability, we could improve the performance of sparse approximations by simply increasing the inducing size. But this is not the case for local aggregations. In terms of robustness, local aggregations are robust to various initializations of hyperparameters due to the local attention mechanism. Finally, we highlight that the proper hybrid of global and local scalable GPs may be a promising way to improve both the model capability and scalability for big data.

When Gaussian Process Meets Big Data: A Review of Scalable GPs

“…For example, though showing a remarkable complexity of O(m 3 ), we cannot expect the subset-of-data to perform well with increasing n. In terms of model capability, global approximations are capable of capturing the global patterns (long-term spatial correlations) but often filter out the local patterns due to the limited global inducing set. In contrast, due to the local nature, local approximations favor capturing local patterns (non-stationary features), enabling them to outperform global approximations for complicated tasks, see the solar example in [38]. The drawback however is that they ignore the global patterns to risk discontinuous predictions and local over-fitting.…”

“…When sharing hyperparameters, the local structure itself may have good estimations of hyperparameter to capture some kind of local patterns[38] 22. Wang et al[157] successfully trained a MVM-based exact GP over a million data points in three days through eight GPUs.…”

A Comparative Assessment of Six Machine Learning Models for Prediction of Bending Force in Hot Strip Rolling Process

“…When GPR is applied to practical problems, GPR can give a confidence interval while outputting the mean value, making the validity of the prediction result continuously enhanced. In addition, because the GPR can quantitatively model Gaussian noise, it has excellent prediction accuracy [48,49]. Because of its good predictive ability, GPR has been widely used in data-driven modeling of various problems in industry [50][51][52][53], so GPR has also become an optional scheme in this paper.…”

Large-Scale Heteroscedastic Regression via Gaussian Process

“…Recently, there has been an increasing trend on the development of scalable GPs, which are classified into two core categories: global approximation and local approximation [25]. As the representative of global approximation, sparse approximation considers m (m ≪ n) global inducing pairs {X m , f m } to optimally summarize the training data by approximating the prior [26] or the posterior [27], resulting in the complexity of O(nm 2 ).…”