Statistical Convergence of the EM Algorithm on Gaussian Mixture Models

Zhao, Ruofei; Li, Yuanzhi; Sun, Yuekai

doi:10.48550/arxiv.1810.04090

Cited by 6 publications

(10 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It guarantees exact recovery as σ → 0. This is the first result showing that the statistical error rate of the EM algorithm does not depend on the distance between any two regression vectors in noisy environment, as opposed to all previous analysis on EM [3,4,8,9,18]. We provide the detailed discussion on this issue in Section 5.…”

Section: Remarkmentioning

confidence: 78%

“…In the special case of two balanced mixtures, global convergence results have been established in [6,7] for GMMs, and in [2] for MLR. Beyond more than two components, a negative result for global convergence of the EM algorithm for 3-GMM has been established [6], while [8,9] give a local convergence result for k-GMM with arbitrary k ≥ 3. Attempts have been made to obtain analogous results for mixed linear regression.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

EM Converges for a Mixture of Many Linear Regressions

Kwon,

Caramanis

2019

Preprint

View full text Add to dashboard Cite

We study the convergence of the Expectation-Maximization (EM) algorithm for mixtures of linear regressions with an arbitrary number k of components. We show that as long as signalto-noise ratio (SNR) is Ω(k), well-initialized EM converges to the true regression parameters. Previous results for k ≥ 3 have only established local convergence for the noiseless setting, i.e., where SNR is infinitely large. Our results enlarge the scope to the environment with noises, and notably, we establish a statistical error rate that is independent of the norm (or pairwise distance) of the regression parameters. In particular, our results imply exact recovery as σ → 0, in contrast to most previous local convergence results for EM, where the statistical error scaled with the norm of parameters. Standard moment-method approaches may be applied to guarantee we are in the region where our local convergence guarantees apply.

show abstract

Section: Remarkmentioning

confidence: 78%

Section: Related Workmentioning

confidence: 99%

EM Converges for a Mixture of Many Linear Regressions

Kwon,

Caramanis

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Results of this nature were then generalized to the k mixture of linear regression in [24]. The k-GM for a general k ≥ 2 was studied in [41,45], which provided results comparable to [3] for gradient EM under minimal separation condition between the means, and closeness of the initial guess to the true means. Beyond the i.i.d.…”

Section: Other Known Resultsmentioning

confidence: 99%

The EM Algorithm is Adaptively-Optimal for Unbalanced Symmetric Gaussian Mixtures

Weinberger,

Bresler

2021

Preprint

View full text Add to dashboard Cite

This paper studies the problem of estimating the means ±θwhere the weights δ * and 1 − δ * are unequal. Assuming that δ * is known, we show that the population version of the EM algorithm globally converges if the initial estimate has non-negative inner product with the mean of the larger weight component. This can be achieved by the trivial initialization θ 0 = 0. For the empirical iteration based on n samples, we show that when initialized at θ 0 = 0, the EM is achieved in no more than O 1 θ * 2 iterations. These results robustify and complement recent results of Wu and Zhou [38] obtained for the equal weights case δ * = 1/2.

show abstract

“…The EM algorithm is a commonly used approach when dealing with latent variables and missing values [49,47,33,16]. The statistical properties, such as the local convergence and minimax optimality of standard EM algorithm has been recently studied in [6,56,60,9,63], while the development of differentially private EM algorithm, especially the theories of the optimal trade-off between privacy and accuracy, is still largely unexplored. In this paper, we propose novel differentially private EM algorithms in both the classic low-dimensional setting and the contemporary high-dimensional setting, where the dimension is much larger than the sample size.…”

Section: Introductionmentioning

confidence: 99%

High-Dimensional Differentially-Private EM Algorithm: Methods and Near-Optimal Statistical Guarantees

Zhang,

Zhang

2021

Preprint

View full text Add to dashboard Cite

In this paper, we develop a general framework to design differentially private expectationmaximization (EM) algorithms in high-dimensional latent variable models, based on the noisy iterative hard-thresholding. We derive the statistical guarantees of the proposed framework and apply it to three specific models: Gaussian mixture, mixture of regression, and regression with missing covariates. In each model, we establish the near-optimal rate of convergence with differential privacy constraints, and show the proposed algorithm is minimax rate optimal up to logarithm factors. The technical tools developed for the high-dimensional setting are then extended to the classic low-dimensional latent variable models, and we propose a near rate-optimal EM algorithm with differential privacy guarantees in this setting. Simulation studies and real data analysis are conducted to support our results.

show abstract

Statistical Convergence of the EM Algorithm on Gaussian Mixture Models

Cited by 6 publications

References 16 publications

EM Converges for a Mixture of Many Linear Regressions

EM Converges for a Mixture of Many Linear Regressions

The EM Algorithm is Adaptively-Optimal for Unbalanced Symmetric Gaussian Mixtures

High-Dimensional Differentially-Private EM Algorithm: Methods and Near-Optimal Statistical Guarantees

Contact Info

Product

Resources

About