Sparse Information Filter for Fast Gaussian Process Regression

Kania, Lucas; Schürch, Manuel; Azzimonti, Dario; Benavoli, Alessio

doi:10.1007/978-3-030-86523-8_32

Cited by 2 publications

(15 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For larger datasets, stochastic optimization has been applied e.g. [2,12,17,27] to obtain faster and more data efficient optimization procedures. For recent reviews on the subject we refer to [20,23,24].…”

Section: Global Sparse Gpsmentioning

confidence: 99%

“…The log of the marginal likelihood of our model formulated in Section 3.3 is L(θ) = log q (y|θ) = log N (0, P ) with P = HS −1 H T + V which can be efficiently computed as detailed in Section A.3 and can be used for deterministic optimization with full batch y for moderate sample size N . However, in order to scale this parameter optimization part to larger number of samples N in a competitive time, stochastic optimization techniques exploiting subsets of data have to be developed similarly done for the global sparse GP model (SVI [12]; REC [27]; IF [17]). We adapt the hybrid approach IF of [17] where we can also exploit an independent factorization of the log marginal likelihood which decomposes into a sum of J terms, so that it can be used for stochastic optimization.…”

Section: Hyperparameter Estimationmentioning

confidence: 99%

“…However, in order to scale this parameter optimization part to larger number of samples N in a competitive time, stochastic optimization techniques exploiting subsets of data have to be developed similarly done for the global sparse GP model (SVI [12]; REC [27]; IF [17]). We adapt the hybrid approach IF of [17] where we can also exploit an independent factorization of the log marginal likelihood which decomposes into a sum of J terms, so that it can be used for stochastic optimization. This constitutes a very fast and accurate alternative for our method as shown in the Appendix A.3 and will also be exploited in Section 4 for large data sets.…”

Section: Hyperparameter Estimationmentioning

confidence: 99%

“…The presented method in the previous section works fine for small datasets, however, in order to scale this parameter optimization part to larger number of samples N in a competitive time, stochastic optimization techniques has to be exploited similarly done for the global sparse GP model (SVI [12]; REC [27]; IF [17]). In the approximation method REC [27], the recursive derivatives are exactly propagated which would also be possible for our model, however, it turned out that in practice the differences in accuracy are very small when using instead the hybrid approach IF of [17]. Thereby, the independent factorization of the log marginal likelihood is used for the computations of the optimization part, whereas the exact posterior is used for inference and prediction.…”

Section: A32 Stochastic Optimizationmentioning

confidence: 99%

“…This optimization procedures were further improved by stochastic optimization e.g. [2,12,17,27], which allows to update the parameters in mini-batches and thus speed up the inference. Optimization of these (variational) parameters helps to scale GP approximations, however, the large number of optimization parameters makes these methods hard to train and they are still limited to M g global inducing points.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Correlated Product of Experts for Sparse Gaussian Process Regression

Schürch¹,

Azzimonti²,

Benavoli³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Gaussian processes (GPs) are an important tool in machine learning and statistics with applications ranging from social and natural science through engineering. They constitute a powerful kernelized non-parametric method with well-calibrated uncertainty estimates, however, off-the-shelf GP inference procedures are limited to datasets with several thousand data points because of their cubic computational complexity. For this reason, many sparse GPs techniques have been developed over the past years. In this paper, we focus on GP regression tasks and propose a new approach based on aggregating predictions from several local and correlated experts. Thereby, the degree of correlation between the experts can vary between independent up to fully correlated experts. The individual predictions of the experts are aggregated taking into account their correlation resulting in consistent uncertainty estimates. Our method recovers independent Product of Experts, sparse GP and full GP in the limiting cases. The presented framework can deal with a general kernel function and multiple variables, and has a time and space complexity which is linear in the number of experts and data samples, which makes our approach highly scalable. We demonstrate superior performance, in a time vs. accuracy sense, of our proposed method against stateof-the-art GP approximation methods for synthetic as well as several real-world datasets with deterministic and stochastic optimization.

show abstract

Section: Global Sparse Gpsmentioning

confidence: 99%

Section: Hyperparameter Estimationmentioning

confidence: 99%

Section: Hyperparameter Estimationmentioning

confidence: 99%

Section: A32 Stochastic Optimizationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Correlated Product of Experts for Sparse Gaussian Process Regression

Schürch¹,

Azzimonti²,

Benavoli³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Correlated product of experts for sparse Gaussian process regression

et al. 2023

Self Cite

View full text Add to dashboard Cite

Gaussian processes (GPs) are an important tool in machine learning and statistics. However, off-the-shelf GP inference procedures are limited to datasets with several thousand data points because of their cubic computational complexity. For this reason, many sparse GPs techniques have been developed over the past years. In this paper, we focus on GP regression tasks and propose a new approach based on aggregating predictions from several local and correlated experts. Thereby, the degree of correlation between the experts can vary between independent up to fully correlated experts. The individual predictions of the experts are aggregated taking into account their correlation resulting in consistent uncertainty estimates. Our method recovers independent Product of Experts, sparse GP and full GP in the limiting cases. The presented framework can deal with a general kernel function and multiple variables, and has a time and space complexity which is linear in the number of experts and data samples, which makes our approach highly scalable. We demonstrate superior performance, in a time vs. accuracy sense, of our proposed method against state-of-the-art GP approximations for synthetic as well as several real-world datasets with deterministic and stochastic optimization.

show abstract

Sparse Information Filter for Fast Gaussian Process Regression

Cited by 2 publications

References 8 publications

Correlated Product of Experts for Sparse Gaussian Process Regression

Correlated Product of Experts for Sparse Gaussian Process Regression

Correlated product of experts for sparse Gaussian process regression

Contact Info

Product

Resources

About