Encoding prior knowledge in the structure of the likelihood

Abstract

The inference of deep hierarchical models is problematic due to strong dependencies between the hierarchies. We investigate a specific transformation of the model parameters based on the multivariate distributional transform. This transformation is a special form of the reparametrization trick, flattens the hierarchy and leads to a standard Gaussian prior on all resulting parameters. The transformation also transfers all the prior information into the structure of the likelihood, hereby decoupling the transformed parameters a priori from each other. A variational Gaussian approximation in this standardized space will be excellent in situations of relatively uninformative data. Additionally, the curvature of the log-posterior is well-conditioned in directions that are weakly constrained by the data, allowing for fast inference in such a scenario. In an example we perform the transformation explicitly for Gaussian process regression with a priori unknown correlation structure. Deep models are inferred rapidly in highly and slowly in poorly informed situations. The flat model show exactly the opposite performance pattern. A synthesis of both, the deep and the flat perspective, provides their combined advantages and overcomes the individual limitations, leading to a faster inference.

Information Field Theory and Artificial Intelligence

“…This paper argues that IFT can be regarded as a specific form of AI by summarizing recent works [29][30][31][32] that suggest this.…”

“…Here, the relation of IFT with artificial intelligence (AI) methods is outlined, in particular with generative neural networks (GNNs). The presented line of arguments summarizes a number of recent works [27][28][29][30][31][32].…”

“…First, the generative models build and used in IFT are GNNs that can interpret data without initial training, thanks to the domain knowledge coded into their architecture [29]. Related architectures have very recently been proposed as image priors in AI as well [44].…”

“…This way, the full model complexity as given by Eqs. 20-25 has been transferred into an effective response function This so called reparametrization trick [41] was introduced to IFT by [29] to simplify numerical VI.…”

Field dynamics inference for local and causal interactions

“…We rely on a variational approach to solve the inference problem, where we approximate the posterior with a Gaussian distribution by minimizing the corresponding forward Kullback-Leibler divergence (KL). Here we follow the approach introduced via [11] and [12]. Introducing the vector θ = (ξ, f, C) into which we include all parameters of the inference, we approximate the posterior via a Gaussian in θ where we also reconstruct the correlation structure of θ including all cross-correlations.…”