Natural Conjugate Gradient in Variational Inference

Abstract

Summary:Variational methods for approximate inference in machine learning often adapt a parametric probability distribution to optimize a given objective function. This view is especially useful when applying variational Bayes (VB) to models outside the conjugate-exponential family. For them, variational Bayesian expectation maximization (VB EM) algorithms are not easily available, and gradientbased methods are often used as alternatives. Traditional natural gradient methods use the Riemannian structure (or geometry) of the predictive distribution to speed up maximum likelihood estimation. We propose using the geometry of the variational approximating distribution instead to speed up a conjugate gradient method for variational learning and inference. The computational overhead is small due to the simplicity of the approximating distribution. Experiments with real-world speech data show significant speedups over alternative learning algorithms. Theory:In previous machine learning algorithms based on natural gradients [1], the aim has been to use maximum likelihood to directly update the model parameters θ taking into account the geometry imposed by the predictive distribution for data p(X|θ). The resulting geometry is often very complicated as the effects of different parameters cannot be separated and the Fisher information matrix is relatively dense.In this paper we propose using natural gradients for free energy minimisation in variational Bayesian learning using the simpler geometry of the approximating distributions q(θ|ξ). Because the approximations are often chosen to minimize dependencies between different parameters θ, the resulting Fisher information matrix with respect to the variational parameters ξ will be mostly diagonal and hence easy to invert.While taking into account the structure of the approximation, plain natural gradient in this case ignores the structure of the model and the global geometry of the parameters θ. This can be addressed by using conjugate gradients. Combining the natural gradient search direction with a conjugate gradient method yields our proposed natural conjugate gradient (NCG) method, which can also be seen as an approximation to the fully Riemannian conjugate gradient method. Experimental results:The NCG algorithm was compared against conjugate gradient (CG) and natural gradient (NG) algorithms in learning a nonlinear state-space model [2]. The results for a number of datasets ranging from 200 to 500 samples of 21 dimensional speech spectrograms can be seen in Figure 1. The plain CG and NG methods were clearly slower than others and the maximum runtime of 24 hours was reached by most CG and some NG runs. NCG was clearly the fastest algorithm with the The results with a larger data set are very similar with NCG outperforming all alternatives by a factor of more than 10. For more details, please refer to the paper. Discussion:The experiments in this paper show that the natural conjugate gradient method outperforms both conjugate gradient and natural gradient methods by a lar...

Variational inference and learning for non-linear state-space models with state-dependent observation noise

“…Variational inference for the model basically follows the scheme introduced in [3,10]: we derive a deterministic approximation of the free energy based on a fixed functional form of the posterior approximation and the apply gradient-based optimisation to minimise the free energy. The optimisation is made more efficient through the use of natural conjugate gradient (NCG) optimisation [11].…”

A gradient-based algorithm competitive with variational Bayesian EM for mixture of Gaussians

“…Our approach, proposed in [6], is that natural gradient could be applied to update both the model parameters and the latent variables by using the geometry of the variational Bayesian approximation q(θ, Z|ξ). The matrix inversion required for the evaluation of the natural gradient in (14) would be prohibitively expensive if the full matrix had to be inverted.…”

“…The matrix inversion required for the evaluation of the natural gradient in (14) would be prohibitively expensive if the full matrix had to be inverted. Luckily, because of the typical factorizing approximation of q, the matrix G is block diagonal [6] without further approximations.…”

“…It was shown to lead to faster convergence and better minima than the standard CG descent in MLP training [4], and to significantly improve the performance in learning a nonlinear state-space model [6].…”

“…In such cases, it is still possible to compute (or at least approximate) the VB cost as a function of the variational parameters ξ and minimize using a suitable optimization method. Recently in [6], such a method was tailored for VB, but it has not been compared to the VB EM algorithm. This paper does the comparison using the mixture of Gaussians (MoG) problem which is known to be well-fitted for the VB EM algorithm.…”