How Deep Are Deep Gaussian Processes?

Abstract

Recent research has shown the potential utility of Deep Gaussian Processes. These deep structures are probability distributions, designed through hierarchical construction, which are conditionally Gaussian. In this paper, the current published body of work is placed in a common framework and, through recursion, several classes of deep Gaussian processes are defined. The resulting samples generated from a deep Gaussian process have a Markovian structure with respect to the depth parameter, and the effective depth of the resulting process is interpreted in terms of the ergodicity, or non-ergodicity, of the resulting Markov chain. For the classes of deep Gaussian processes introduced, we provide results concerning their ergodicity and hence their effective depth. We also demonstrate how these processes may be used for inference; in particular we show how a Metropolis-within-Gibbs construction across the levels of the hierarchy can be used to derive sampling tools which are robust

Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo

“…Unlike with VI, when using sampling methods, we do not have access to an approximate posterior distribution q(u) to generate predictions with. Instead, we have to rely on approximate samples generated from the posterior which in turn can be used to make predictions [Dunlop et al, 2017, Hoffman, 2017.…”