Acceleration schemes with application to the EM algorithm

Berlinet, Alain; Roland, Ch.

doi:10.1016/j.csda.2006.12.013

Cited by 19 publications

(16 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For this purpose, we recall in Section 2 the definition of the polynomial vector extrapolation methods of order k. The choice is motivated by the fact that these methods which combine successive Picard's iterates, are powerful methods to solve the problem (1) and that the Lemaréchal's method is one of the simplest members of this family. In Section 3, we justify the construction of the adjusted method by means of a new polynomial extrapolation transformation and we define the squaring technique for any order k. Moreover, we study the required operation count and storage.…”

Section: It Is Described Bymentioning

confidence: 99%

Squared polynomial extrapolation methods with cycling: an application to the positron emission tomography problem

2007

Self Cite

View full text Add to dashboard Cite

Roland and Varadhan (Appl. Numer. Math., 55:215-226, 2005) presented a new idea called "squaring" to improve the convergence of Lemaréchal's scheme for solving nonlinear fixed-point problems. Varadhan and Roland (Squared extrapolation methods: A new class of simple and efficient numerical schemes for accelerating the convergence of the EM algorithm, Department of Biostatistics Working Paper. Johns Hopkins University, http://www.bepress.com/jhubiostat/paper63, 2004) noted that Lemaréchal's scheme can be viewed as a member of the class of polynomial extrapolation methods with cycling that uses two fixed-point iterations per cycle. Here we combine these two ideas, cycled extrapolation and squaring, and construct a new class of methods, called squared polynomial methods (SQUAREM), for accelerating the convergence of fixed-point iterations. Our main goal is to evaluate whether the squaring device is effective in improving the rate of convergence of cycled extrapolation methods that use more than two fixed- 160 Numer Algor (2007) 44:159-172 point iterations per cycle. We study the behavior of the new schemes on an image reconstruction problem for positron emission tomography (PET) using simulated data. Our numerical experiments show the effectiveness of first-and higher-order squared polynomial extrapolation methods in accelerating image reconstruction, and also their relative superiority compared to the classical, "unsquared" vector polynomial methods.

show abstract

Section: It Is Described Bymentioning

confidence: 99%

Squared polynomial extrapolation methods with cycling: an application to the positron emission tomography problem

2007

Self Cite

View full text Add to dashboard Cite

show abstract

“…We initially investigated a variety of schemes to accelerate the EM iterations along the lines of Varadhan and Roland (2008) and Berlinet and Roland (2007), which while somewhat helpful did not significantly improve the computational speed. The crucial insight came again only with the realization that the task of maximizing L(f) is a convex problem.…”

Section: Non-parametric Maximum Likelihoodmentioning

confidence: 99%

Convex Optimization, Shape Constraints, Compound Decisions, and Empirical Bayes Rules

Koenker¹,

Mizera²

2014

Journal of the American Statistical Association

159

246

View full text Add to dashboard Cite

Abstract. Estimation of mixture densities for the classical Gaussian compound decision problem and their associated (empirical) Bayes rules is considered from two new perspectives. The first, motivated by Brown and Greenshtein (2009), introduces a nonparametric maximum likelihood estimator of the mixture density subject to a monotonicity constraint on the resulting Bayes rule. The second, motivated by Jiang and Zhang (2009), proposes a new approach to computing the Kiefer-Wolfowitz nonparametric maximum likelihood estimator for mixtures. In contrast to prior methods for these problems, our new approaches are cast as convex optimization problems that can be efficiently solved by modern interior point methods. In particular, we show that the reformulation of the Kiefer-Wolfowitz estimator as a convex optimization problem reduces the computational effort by several orders of magnitude for typical problems, by comparison to prior EM-algorithm based methods, and thus greatly expands the practical applicability of the resulting methods. Our new procedures are compared with several existing empirical Bayes methods in simulations employing the well-established design of Johnstone and Silverman (2004). Some further comparisons are made based on prediction of baseball batting averages. A Bernoulli mixture application is briefly considered in the penultimate section.

show abstract

“…Previously, Huang et al (2005) have reported that CTJEM converged faster than the adaptive overrelaxed EM (aEM) by for the Bayesian networks when data is highly sparse. More recently, we conducted a comprehensive empirical comparison of several new EM acceleration methods published from 2003 to 2007 Varadhan and Roland 2004;Kuroda and Sakakihara 2006;Berlinet and Roland 2007). We found that aEM still outperformed other new methods, only second to TJ 2 aEM, a variant of TJEM that we developed to accelerate aEM (Huang et al 2007a).…”

Section: Bayesian Networkmentioning

confidence: 87%

Global and componentwise extrapolations for accelerating training of Bayesian networks and conditional random fields

Huang

Yang

Chang

et al. 2009

Data Min Knowl Disc

View full text Add to dashboard Cite

The triple jump extrapolation method is an effective approximation of Aitken's acceleration that can accelerate the convergence of many algorithms for data mining, including EM and generalized iterative scaling (GIS). It has two optionsglobal and componentwise extrapolation. Empirical studies showed that neither can dominate the other and it is not known which one is better under what condition. In this paper, we investigate this problem and conclude that, when the Jacobian is (block) diagonal, componentwise extrapolation will be more effective. We derive two hints to determine the block diagonality. The first hint is that when we have a highly sparse data set, the Jacobian of the EM mapping for training a Bayesian network will be block diagonal. The second is that the block diagonality of the Jacobian of the GIS mapping for training CRF is negatively correlated with the strength of feature dependencies. We empirically verify these hints with controlled and real-world data sets and show that our hints can accurately predict which method will be superior. We also show that both global and componentwise extrapolation can provide substantial acceleration. In particular, when applied to train large-scale CRF models, the GIS variant accelerated 123 Global and componentwise extrapolations 59 by componentwise extrapolation not only outperforms its global extrapolation counterpart, as our hint predicts, but can also compete with limited-memory BFGS (L-BFGS), the de facto standard for CRF training, in terms of both computational efficiency and F-scores. Though none of the above methods are as fast as stochastic gradient descent (SGD), careful tuning is required for SGD and the results given in this paper provide a useful foundation for automatic tuning.

show abstract

Acceleration schemes with application to the EM algorithm

Cited by 19 publications

References 5 publications

Squared polynomial extrapolation methods with cycling: an application to the positron emission tomography problem

Squared polynomial extrapolation methods with cycling: an application to the positron emission tomography problem

Convex Optimization, Shape Constraints, Compound Decisions, and Empirical Bayes Rules

Global and componentwise extrapolations for accelerating training of Bayesian networks and conditional random fields

Contact Info

Product

Resources

About