Scalable Cross Validation Losses for Gaussian Process Models

Jankowiak, Martin; Pleiss, Geoff

doi:10.48550/arxiv.2105.11535

Cited by 2 publications

(4 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This supports our numerical and theoretical evidence of the advantages of nearby sampling in Section 3.3. Third, Vecchia does perform well overall in terms of prediction performance, which is contrary to the finding in Jankowiak and Pleiss (2021). Most likely, the heuristic MMD ordering we adopted offers significant improvement in model approximation over the default coordinate-based ordering (Guinness, 2018).…”

Section: Case Studiesmentioning

confidence: 66%

Gaussian Process Inference Using Mini-batch Stochastic Gradient Descent: Convergence Guarantees and Empirical Benefits

Chen¹,

Zheng²,

Kontar³

et al. 2021

Preprint

View full text Add to dashboard Cite

Stochastic gradient descent (SGD) and its variants have established themselves as the go-to algorithms for large-scale machine learning problems with independent samples due to their generalization performance and intrinsic computational advantage. However, the fact that the stochastic gradient is a biased estimator of the full gradient with correlated samples has led to the lack of theoretical understanding of how SGD behaves under correlated settings and hindered its use in such cases. In this paper, we focus on hyperparameter estimation for the Gaussian process (GP) and take a step forward towards breaking the barrier by proving minibatch SGD converges to a critical point of the full log-likelihood loss function, and recovers model hyperparameters with rate O( 1 K ) for K iterations, up to a statistical error term depending on the minibatch size. Our theoretical guarantees hold provided that the kernel functions exhibit exponential or polynomial eigendecay which is satisfied by a wide range of kernels commonly used in GPs. Numerical studies on both simulated and real datasets demonstrate that minibatch SGD has better generalization over state-of-the-art GP methods while reducing the computational burden and opening a new, previously unexplored, data size regime for GPs.

show abstract

Section: Case Studiesmentioning

confidence: 66%

Gaussian Process Inference Using Mini-batch Stochastic Gradient Descent: Convergence Guarantees and Empirical Benefits

Chen¹,

Zheng²,

Kontar³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…. This is because F's preference for sparsity is heavily influenced by hyperparameter c, as there is a fixed variable inclusion cost of log( 1c ) (see the appendix), making model weights overly dependent on c. Moreover, the LOO-LPD is more robust against overfitting and model mis-specification than the log evidence (Jankowiak and Pleiss, 2021;Gelman et al, 2014).…”

Section: Approximating Posterior Model Weights With Leave-one-out Log...mentioning

confidence: 99%

“…To compute the LOO-LPD in efficient O(n 3 ) time in small datasets we use the Bürkner et al (2021) algorithm, which uses rank 1 updates to K XX . In large datasets we nearest-neighbour truncate the LOO-LPD following the success of this approach in Jankowiak and Pleiss (2021). That is, for each (y i , x i ), we condition on m nearest neighbours of x i found using the distance metric in the kernel.…”

Section: Approximating Posterior Model Weights With Leave-one-out Log...mentioning

confidence: 99%

“…We test on n * = 10 4 datapoints but with x 1 , x 2 grid spaced on [0, 1]. We implement the SGP and SVGP using GPyTorch (Gardner et al, 2018) with 512 and 1024 inducing points respectively consistent with standard practice (Chen et al, 2020;Wang et al, 2019;Jankowiak and Pleiss, 2021). We implement the SSVGP with m = {64, 128, 256} minibatch sizes, and use 64 and 256 neighbours in LOO-LPD and predictive distribution truncation respectively.…”

Section: Experiments 2: a Large-scale Sparse Prediction Challengementioning

confidence: 99%

See 1 more Smart Citation

Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes

Dance¹,

Paige²

2021

Preprint

View full text Add to dashboard Cite

Variable selection in Gaussian processes (GPs) is typically undertaken by thresholding the inverse lengthscales of 'automatic relevance determination' kernels, but in highdimensional datasets this approach can be unreliable. A more probabilistically principled alternative is to use spike and slab priors and infer a posterior probability of variable inclusion. However, existing implementations in GPs are extremely costly to run in both high-dimensional and large-n datasets, or are intractable for most kernels. As such, we develop a fast and scalable variational inference algorithm for the spike and slab GP that is tractable with arbitrary differentiable kernels. We improve our algorithm's ability to adapt to the sparsity of relevant variables by Bayesian model averaging over hyperparameters, and achieve substantial speed ups using zero temperature posterior restrictions, dropout pruning and nearest neighbour minibatching. In experiments our method consistently outperforms vanilla and sparse variational GPs whilst retaining similar runtimes (even when n = 10 6 ) and performs competitively with a spike and slab GP using MCMC but runs up to 1000 times faster.

show abstract

Scalable Cross Validation Losses for Gaussian Process Models

Cited by 2 publications

References 14 publications

Gaussian Process Inference Using Mini-batch Stochastic Gradient Descent: Convergence Guarantees and Empirical Benefits

Gaussian Process Inference Using Mini-batch Stochastic Gradient Descent: Convergence Guarantees and Empirical Benefits

Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes

Contact Info

Product

Resources

About