On EM algorithms and their proximal generalizations

Abstract

Abstract.In this paper, we analyze the celebrated EM algorithm from the point of view of proximal point algorithms. More precisely, we study a new type of generalization of the EM procedure introduced in [4] and called Kullback-proximal algorithms. The proximal framework allows us to prove new results concerning the cluster points. An essential contribution is a detailed analysis of the case where some cluster points lie on the boundary of the parameter space.Résumé. Le but de cet article est de proposer une analyse de l'algorithme EM du point de vue des algorithmes de point proximal. Nousétudions plus précisément un nouveau type de procedure EM généralisée introduit dans [4] et appelé algorithme Kullback proximal. Le cadre des algorithmes proximaux nous permet d'obtenir de nouveaux résultats sur les points d'accumulation. Une contribution essentielle est l'analyse détaillée du cas où un point d'accumulation est sur le bord du domaine des paramètres.

A Proximal Point Algorithm for Minimum Divergence Estimators with Application to Mixture Models

“…It is not empty since Φ 0 is compact. The remaining of the proof is a direct result of Theorem 3.3.1 from [18]. The strict concavity of the objective function around an accumulation point is replaced here by the strict convexity of the estimated divergence.…”

“…In these results, assumption A3 is essential. Although in [18] this problem is avoided, their approach demands that the log-likelihood has −∞ limit as φ → ∞. This is simply not verified for mixture models.…”

“…This is simply not verified for mixture models. We present a similar method to the one in [18] based on the idea of Tseng [2] of using the set Φ 0 which is valid for mixtures. We lose, however, the guarantee of consecutive decrease of the sequence (φ k ) k .…”

A proximal approach for optimization problems involving kullback divergences

“…, e S,n ∈ R KS correspond to some possible additive summable errors in the computation of the proximity operators. If the set of solutions to Problem 1.1 is nonempty, then any sequence (x n ) n∈N generated by Algorithm (12) converges to an element of this set (under a suitable weak qualification condition). Note that this algorithm has two interesting features.…”

“…However, the required optimization steps may be difficult to implement and the convergence of the algorithm is only guaranteed under restrictive conditions. Moreover, a Kullback-proximal algorithm generalizing the EM algorithm was investigated in [12]. The KL divergence is then used as a metric for the maximization of a log-likelihood function, rather than being one of the terms of the objective function.…”