Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo

Havasi, Marton; Hernández-Lobato, José Miguel; Murillo-Fuentes, Juan José

doi:10.48550/arxiv.1806.05490

“…Moreover, our experiments involving more complicated architectures, such as ResNet or DenseNet for standard multi-class classification, have not managed to surpass in accuracy a far less complex model with only 3 hidden layers. A plausible reason behind this under-fitting resides in the factorized approximate posterior formulation, which was shown to negatively affect predictive performance compared to MCMC inference schemes (Havasi et al, 2018). We posit that using alternative inference frameworks (Ustyuzhaninov et al, 2019) whereby we impose correlations between layers might alleviate this issue.…”

Section: Discussionmentioning

confidence: 99%

Distributional Gaussian Processes Layers for Out-of-Distribution Detection

Popescu

¹

,

Sharp

²

,

Cole

³

et al. 2022

Melba

0

View full text Add to dashboard Cite

Machine learning models deployed on medical imaging tasks must be equipped with out-of-distribution detection capabilities in order to avoid erroneous predictions. It is unsure whether out-of-distribution detection models reliant on deep neural networks are suitable for detecting domain shifts in medical imaging. Gaussian Processes can reliably separate in-distribution data points from out-of-distribution data points via their mathematical construction. Hence, we propose a parameter efficient Bayesian layer for hierarchical convolutional Gaussian Processes that incorporates Gaussian Processes operating in Wasserstein-2 space to reliably propagate uncertainty. This directly replaces convolving Gaussian Processes with a distance-preserving affine operator on distributions. Our experiments on brain tissue-segmentation show that the resulting architecture approaches the performance of well-established deterministic segmentation algorithms (U-Net), which has not been achieved with previous hierarchical Gaussian Processes. Moreover, by applying the same segmentation model to out-of-distribution data (i.e., images with pathology such as brain tumors), we show that our uncertainty estimates result in out-of-distribution detection that outperforms the capabilities of previous Bayesian networks and reconstruction-based approaches that learn normative distributions.

show abstract

“…Moreover, our experiments involving more complicated architectures, such as ResNet or DenseNet for standard multi-class classification, have not managed to surpass in accuracy a far less complex model with only 3 hidden layers. A plausible reason behind this under-fitting resides in the factorized approximate posterior formulation, which was shown to negatively affect predictive performance compared to MCMC inference schemes (Havasi et al, 2018). We posit that using alternative inference frameworks (Ustyuzhaninov et al, 2019) whereby we impose correlations between layers might alleviate this issue.…”

Section: Discussionmentioning

confidence: 99%

Untitled

2022

Melba

0

View full text Add to dashboard Cite

Machine learning models deployed on medical imaging tasks must be equipped with out-of-distribution detection capabilities in order to avoid erroneous predictions. It is unsure whether out-of-distribution detection models reliant on deep neural networks are suitable for detecting domain shifts in medical imaging. Gaussian Processes can reliably separate in-distribution data points from out-of-distribution data points via their mathematical construction. Hence, we propose a parameter efficient Bayesian layer for hierarchical convolutional Gaussian Processes that incorporates Gaussian Processes operating in Wasserstein-2 space to reliably propagate uncertainty. This directly replaces convolving Gaussian Processes with a distance-preserving affine operator on distributions. Our experiments on brain tissue-segmentation show that the resulting architecture approaches the performance of well-established deterministic segmentation algorithms (U-Net), which has not been achieved with previous hierarchical Gaussian Processes. Moreover, by applying the same segmentation model to out-of-distribution data (i.e., images with pathology such as brain tumors), we show that our uncertainty estimates result in out-of-distribution detection that outperforms the capabilities of previous Bayesian networks and reconstruction-based approaches that learn normative distributions. To facilitate future work our code is publicly available 1 .

show abstract

“…e Stochastic Gradient Hamiltonian Monte Carlo approach (Ma et al, 2015) has proven e cient in deep GPs (Havasi et al, 2018) and in GANs (Saatci and Wilson, 2017). Another avenue for improvement lies in kernel interpolation techniques (Wilson and Nickisch, 2015;Evans and Nair, 2018) which would make inference and prediction faster.…”

Section: Discussionmentioning

confidence: 99%

Deep Convolutional Gaussian Processes

Blomqvist

¹

,

Kaski

²

,

Heinonen

³

2020

Machine Learning and Knowledge Discovery in Databases

View full text Add to dashboard Cite

We propose deep convolutional Gaussian processes, a deep Gaussian process architecture with convolutional structure.e model is a principled Bayesian framework for detecting hierarchical combinations of local features for image classication. We demonstrate greatly improved image classication performance compared to current Gaussian process approaches on the MNIST and CIFAR-10 datasets. In particular, we improve CIFAR-10 accuracy by over 10 percentage points.

show abstract

Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo

Cited by 6 publications

References 6 publications

Distributional Gaussian Processes Layers for Out-of-Distribution Detection

Distributional Gaussian Processes Layers for Out-of-Distribution Detection

Untitled

Deep Convolutional Gaussian Processes

Contact Info

Product

Resources

About