A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants

Neal, Radford M.; Hinton, Geoffrey E.

doi:10.1007/978-94-011-5014-9_12

Cited by 1,535 publications

(1,404 citation statements)

References 3 publications

Supporting

Mentioning

1,379

Contrasting

Unclassified

Order By: Relevance

“…Initially (Hinton and von Cramp, 1993;MacKay, 1995a,b) the free energy was described in terms of description lengths and coding. Later, established methods like EM were considered in the light of variational free energy (Neal and Hinton, 1998; see also Bishop, 1999). Variational learning can be regarded as subsuming most other learning schemes as special cases.…”

Section: The Variational Approachmentioning

confidence: 99%

Variational free energy and the Laplace approximation

et al. 2007

View full text Add to dashboard Cite

This note derives the variational free energy under the Laplace approximation, with a focus on accounting for additional model complexity induced by increasing the number of model parameters. This is relevant when using the free energy as an approximation to the log-evidence in Bayesian model averaging and selection. By setting restricted maximum likelihood (ReML) in the larger context of variational learning and expectation maximisation (EM), we show how the ReML objective function can be adjusted to provide an approximation to the log-evidence for a particular model. This means ReML can be used for model selection, specifically to select or compare models with different covariance components. This is useful in the context of hierarchical models because it enables a principled selection of priors that, under simple hyperpriors, can be used for automatic model selection and relevance determination (ARD). Deriving the ReML objective function, from basic variational principles, discloses the simple relationships among Variational Bayes, EM and ReML. Furthermore, we show that EM is formally identical to a full variational treatment when the precisions are linear in the hyperparameters. Finally, we also consider, briefly, dynamic models and how these inform the regularisation of free energy ascent schemes, like EM and ReML.

show abstract

Section: The Variational Approachmentioning

confidence: 99%

Variational free energy and the Laplace approximation

et al. 2007

View full text Add to dashboard Cite

show abstract

“…In order for this convergence property to apply, it is necessary that the bound h(θ|θ k ) touches the objective function. Previous work has, however, shown that 'approximate' EM can be effective in practice despite lacking an analytic convergence guarantee [14], [19]. In the following sections we show that exact EM is intractable in the hybrid model learning problem, but propose a tractable, approximate EM approach that is effective in practice.…”

Section: Review Of Expectation-maximizationmentioning

confidence: 84%

Model learning for switching linear systems with autonomous mode transitions

Blackmore

Gil

Chung

et al. 2007

2007 46th IEEE Conference on Decision and Control

View full text Add to dashboard Cite

Abstract-We present a novel method for model learning in hybrid discrete-continuous systems. The approach uses approximate Expectation-Maximization to learn the MaximumLikelihood parameters of a switching linear system. The approach extends previous work by 1) considering autonomous mode transitions, where the discrete transitions are conditioned on the continuous state, and 2) learning the effects of control inputs on the system. We evaluate the approach in simulation.

show abstract

“…The E-step can be viewed as the search over the space of distributions q of the latent variables W, keeping the parameters Θ fixed (3.7), and the M-step can be interpreted to be the search over the parameter space, keeping the latent variables distribution q fixed (3.8). The cost function for the EM is given by [15]:…”

Section: Inference For Probabilistic Pca Via Variational Emmentioning

confidence: 99%

A Deflation Method for Structured Probabilistic PCA

Khanna

Ghosh

Poldrack

et al. 2017

Proceedings of the 2017 SIAM International Conference on Data Mining

View full text Add to dashboard Cite

Modern treatments of structured Principal Component Analysis often focus on the estimation of a single component under various assumptions or priors, such as sparsity and smoothness, and then the procedure is extended to multiple components by sequential estimation interleaved with deflation. While prior work has highlighted the importance of proper deflation for ensuring the quality of the estimated components, to our knowledge, proposed techniques have only been developed and applied to non-probabilistic principal component analyses, and are not trivially extended to probabilistic analyses. This work introduces a novel, robust and efficient deflation method for Probabilistic Principal Component Analysis using tools recently developed for constrained probabilistic estimation via information projection. The components estimated using the proposed deflation regain some of the interpretability of classic PCA such as straightforward estimates of variance explained, while retaining the ability to incorporate rich prior structure. Moreover, sequential estimation allows for scaling probabilistic techniques to be at par with their deterministic counterparts. Experimental results on simulated data demonstrate the utility of the proposed deflation in terms of component recovery, and evaluation on neuroimaging data show both qualitative and quantitative improvements in the quality of the estimated components. We also present timing experiments on real data to illustrate the importance of sequential estimation with proper deflation for scalability.

show abstract

A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants

Cited by 1,535 publications

References 3 publications

Variational free energy and the Laplace approximation

Variational free energy and the Laplace approximation

Model learning for switching linear systems with autonomous mode transitions

A Deflation Method for Structured Probabilistic PCA

Contact Info

Product

Resources

About