Random Feature Expansions for Deep Gaussian Processes

Cutajar, Kurt; Bonilla, Edwin V.; Michiardi, Pietro; Filippone, Maurizio

doi:10.48550/arxiv.1610.04386

Cited by 3 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As a consequence, there is little benefit in adding additional layers after a certain point. This observation elucidates the mechanism underlying the choices of DGPs with a small number of layers for inference in numerous papers, for example in Cutajar et al (2016); Salimbeni and Deisenroth (2017); Dai et al (2015).…”

Section: Our Contributionsupporting

confidence: 63%

See 1 more Smart Citation

How Deep Are Deep Gaussian Processes?

Dunlop,

Girolami,

Stuart

et al. 2017

Preprint

View full text Add to dashboard Cite

Recent research has shown the potential utility of Deep Gaussian Processes. These deep structures are probability distributions, designed through hierarchical construction, which are conditionally Gaussian. In this paper, the current published body of work is placed in a common framework and, through recursion, several classes of deep Gaussian processes are defined. The resulting samples generated from a deep Gaussian process have a Markovian structure with respect to the depth parameter, and the effective depth of the resulting process is interpreted in terms of the ergodicity, or non-ergodicity, of the resulting Markov chain. For the classes of deep Gaussian processes introduced, we provide results concerning their ergodicity and hence their effective depth. We also demonstrate how these processes may be used for inference; in particular we show how a Metropolis-within-Gibbs construction across the levels of the hierarchy can be used to derive sampling tools which are robust

show abstract

Section: Our Contributionsupporting

confidence: 63%

“…3. Following Neal (1995);Duvenaud et al (2014), recent works such as Dai et al (2015); Cutajar et al (2016) connect all layers to the input layer in order to avoid certain pathologies. The Markovian structure of the process is maintained in this case: with the above notation, the process is then defined by…”

Section: Proofmentioning

confidence: 99%

How Deep Are Deep Gaussian Processes?

Dunlop,

Girolami,

Stuart

et al. 2017

Preprint

View full text Add to dashboard Cite

show abstract

“…In this case, the gradient mixing effect will also be noticeable. We hypothesize that it could be part of the reason of the pathologies in weight-space methods; as a supporting evidence, we experimented SVGD on the random feature approximation to deep GP(Cutajar et al, 2016) on the synthetic dataset fromLouizos & Welling (2016). Using a 3-layer deep GP with Arc kernel and 5 GPs per layer, all particles collapse to a constant function, which is not the posterior mean given by HMC.…”

mentioning

confidence: 92%

Function Space Particle Optimization for Bayesian Neural Networks

Wang¹,

Ren²,

Zhu³

et al. 2019

Preprint

View full text Add to dashboard Cite

While Bayesian neural networks (BNNs) have drawn increasing attention, their posterior inference remains challenging, due to the high-dimensional and overparameterized nature. Recently, several highly flexible and scalable variational inference procedures based on the idea of particle optimization have been proposed. These methods directly optimize a set of particles to approximate the target posterior. However, their application to BNNs often yields sub-optimal performance, as they have a particular failure mode on over-parameterized models. In this paper, we propose to solve this issue by performing particle optimization directly in the space of regression functions. We demonstrate through extensive experiments that our method successfully overcomes this issue, and outperforms strong baselines in a variety of tasks including prediction, defense against adversarial examples, and reinforcement learning. 1 * corresponding author 1 This version extends the ICLR paper with results connecting the final algorithm to Wasserstein gradient flows.

show abstract

“…Deep Gaussian Processes (DGP) [Damianou and Lawrence, 2013] are multilayer predictive models that are highly flexible and can accurately model uncertainty. In particular, they have been shown to perform well on a multitude of supervised regression tasks ranging from small (∼500 datapoints) to large datasets (∼500,000 datapoints) [Salimbeni and Deisenroth, 2017, Bui et al, 2016, Cutajar et al, 2016. Their main benefit over neural networks is that they are capable of capturing uncertainty in their predictions.…”

Section: Introductionmentioning

confidence: 99%

Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo

Havasi,

Hernández-Lobato,

Murillo-Fuentes

2018

Preprint

View full text Add to dashboard Cite

Deep Gaussian Processes (DGPs) are hierarchical generalizations of Gaussian Processes that combine well calibrated uncertainty estimates with the high flexibility of multilayer models. One of the biggest challenges with these models is that exact inference is intractable. The current state-of-the-art inference method, Variational Inference (VI), employs a Gaussian approximation to the posterior distribution. This can be a potentially poor unimodal approximation of the generally multimodal posterior. In this work, we provide evidence for the non-Gaussian nature of the posterior and we apply the Stochastic Gradient Hamiltonian Monte Carlo method to generate samples. To efficiently optimize the hyperparameters, we introduce the Moving Window MCEM algorithm. This results in significantly better predictions at a lower computational cost than its VI counterpart. Thus our method establishes a new state-of-the-art for inference in DGPs.32nd Conference on Neural Information Processing Systems (NIPS 2018),

show abstract

Random Feature Expansions for Deep Gaussian Processes

Cited by 3 publications

References 0 publications

How Deep Are Deep Gaussian Processes?

How Deep Are Deep Gaussian Processes?

Function Space Particle Optimization for Bayesian Neural Networks

Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo

Contact Info

Product

Resources

About