Deep Gaussian Covariance Network

Abstract

The correlation length-scale θ next to the noise variance σ 2 are the most used hyperparameters for the Gaussian processes GP. Typically stationary covariance functions k(x i , x j ) are used, which are only dependent on the distances between input points τ = ||x i − x j || and thus invariant to the translations in the input space X. The optimization of the hyperparameters is commonly done by maximizing the log marginal likelihood log p(y|X, θ, σ 2 ). This works quite well, if τ ∼ U(0, max(τ )), since θ fits for every subspace in X. In the case of τ ∼ U(0, max(τ )) like in a locally adapted or even sparse X, the prediction ŷGP (x * ) of a test point x * can be worse dependent of its position. A possible solution to this, is the usage of a non-stationary covariance function k(τ * , θ * ), where θ * is calculated by a deep neural network dnn(x * ). So that the correlation length scales θ * and also possibly the noise variance σ 2 * are dependent on the test point x * . Furthermore, different types of covariance functions are trained simultaneously, so that the GP prediction is an additive overlay of different covariance matrices ŷGP (X * ) = K 1 (X, X * , θ 1 (X * ), σ 2 1 (X * )) + ... + K n (X, X * , θ n (X * ), σ 2 n (X * )). The right covariance functions combination and its hyperparameters θ(X * ), σ 2 (X * ) are learned by the deep neural network. Additional, the GP will be able to be trained by batches or online and so it can handle arbitrarily large data sets. We call this framework Deep Gaussian Covariance Network (DGCN). There are also further extensions to this framework possible, for example sequentially dependent problems like time series or the local mixture of experts. The basic framework and some extension possibilities will be presented in this work. Moreover, a comparison to some recent state of the art surrogate model methods will be performed, also for a time dependent problem.

When Gaussian Process Meets Big Data: A Review of Scalable GPs

“…The NNs+GP structure produces sensible uncertainties, and is found to be robust to adversarial examples in CV tasks [191]. Particularly, Cremanns and Roos [192] employed the same hybrid structure, but used the NNs to learn input-dependent hyperparameters for the additive kernels. Then, the NeNe algorithm is employed to ease the GP inference.…”

Gradient and uncertainty enhanced sequential sampling for global fit

“…A GP model with posterior mean function fe LOO with Matérn kernel (ν = 3/2) was used to approximate LOOCV error in Eq. (12). Moreover, Delaunay triangulation [55] was used to select the support points in Eq.…”

“…Various extensions to GPs such as the deep GPs [8], sparse GPs based on variational inference [9,10], and efficient matrix decomposition [11] were developed to overcome some of these limitations. One particularly promising approach is the combination of GPs with Artificial Neural Networks (ANNs) to Deep Gaussian Covariance Networks (DGCNs) [12,13] to learn the non-stationary hyperparameters of the GP together with combinations of different covariance functions.…”

Fast parameter estimation of generalized extreme value distribution using neural networks

“…Motivated by recent advancements in applying deep learning algorithms to extreme events, as discussed earlier, and inspired by the successful utilization of NNs for time series and spatial data, as evidenced in papers such as Cremanns and Roos (2017), Gerber and Nychka (2021), Majumder et al (2022), and Wikle and Zammit-Mangion (2023), our research introduces a novel estimation method. In this work, we present a new estimation method that utilizes a deep NN to fit univariate GEV distributions to extreme events.…”