Recent Advances in Scaling Up Gaussian Process Predictive Models for Large Spatiotemporal Data

Abstract

Abstract. The expressive power of Gaussian process (GP) models comes at a cost of poor scalability in the size of the data. To improve their scalability, this paper presents an overview of our recent progress in scaling up GP models for large spatiotemporally correlated data through parallelization on clusters of machines, online learning, and nonmyopic active sensing/learning.

Private Outsourced Bayesian Optimization

“…We will also consider our outsourced setting in the active learning context (Cao et al, 2013;Hoang et al, 2014a;Low et al, 2008;2009;Ouyang et al, 2014;Zhang et al, 2016). For applications with a huge budget of function evaluations, we like to couple PO-GP-UCB with the use of distributed/decentralized (Chen et al, 2012;2013a;Hoang et al, 2016;2019b;a;Low et al, 2015;Ouyang & Low, 2018) or online/stochastic (Hoang et al, 2015;Low et al, 2014b;Xu et al, 2014;Teng et al, 2020;Yu et al, 2019a;…”

Stochastic Variational Inference for Bayesian Sparse Gaussian Process Regression

“…As a result, they incur linear time in the data size that is still prohibitively expensive for training with big data (i.e., million-sized datasets). To scale up to big data, parallel [3]- [5] and online [6], [7] variants of several of these SGPR models have been developed for prediction (by assuming known hyperparameters) but not hyperparameter learning.…”

Collective Online Learning of Gaussian Processes in Massive Multi-Agent Systems

“…However, such algorithms fall short of achieving the truly decentralized GP fusion necessary for scaling up to a massive number of agents grounded in the real world (e.g., traffic sensing, modeling, and prediction by autonomous vehicles cruising in urban road networks (Chen et al 2015;Low et al 2015a;Hoang et al 2014;Min and Wynter 2011;Ouyang et al 2014;Wang and Papageorgiou 2005;Work et al 2010), distributed inference on a network of IoTs, surveillance cameras and mobile devices/robots (Kang and Larkin 2016;Natarajan et al 2014;Hoang et al 2018b;Zhang et al 2016)) due to the following critical issues: (a) An obvious limitation is the single point(s) of failure with the server agent(s) whose computational and communication capabilities must be superior and robust (e.g., against transmission loss); (b) different GP inference agents are likely to gather data of varying behaviors and correlation structure from possibly separate localities of the input domain (e.g., spatiotemporal) and would therefore incur considerable information loss due to summarization based on a common set of fixed/known GP hyperparameter settings and inducing inputs, especially when the inducing inputs are few and far from the data (in the correlation sense); and (c) like distributed GP models, distributed GP fusion algorithms implicitly assume a one-time processing of a fixed set of data and would hence repeat the entire fusion process involving all local data gathered by the agents whenever new batches of streaming data arrive, which is prohibitively expensive. To overcome these limitations, this paper presents a novel Collective Online Learning of GPs (COOL-GP) framework for enabling a massive number of agents to simultaneously perform (a) efficient online updates of their GP models using their local streaming data with varying correlation structures and (b) decentralized fusion of their resulting online GP models with different learned hyperparameter settings and inducing inputs residing in the original input domain.…”