EM Converges for a Mixture of Many Linear Regressions

Abstract

We study the convergence of the Expectation-Maximization (EM) algorithm for mixtures of linear regressions with an arbitrary number k of components. We show that as long as signalto-noise ratio (SNR) is Ω(k), well-initialized EM converges to the true regression parameters. Previous results for k ≥ 3 have only established local convergence for the noiseless setting, i.e., where SNR is infinitely large. Our results enlarge the scope to the environment with noises, and notably, we establish a statistical error rate that is independent of the norm (or pairwise distance) of the regression parameters. In particular, our results imply exact recovery as σ → 0, in contrast to most previous local convergence results for EM, where the statistical error scaled with the norm of parameters. Standard moment-method approaches may be applied to guarantee we are in the region where our local convergence guarantees apply.

Learning Mixtures of Linear Regressions in Subexponential Time via Fourier Moments

“…In particular, if ∆ = Ω(1), we again attain runtime which is sub-exponential in k. In the special case when the mixing weights are all known, and assuming that ς = O( ∆ k 2 polylog(k) ), by combining this result with the local convergence result of [KC19], we can again attain arbitrarily good accuracy by slightly increasing the runtime; see Section 8.5 and Theorem 8.33 for details.…”

“…In Section 8.4 we prove Theorem 8.1. Finally, in Section 8.5, we briefly describe how to leverage the local convergence result of [KC19] in conjunction with our algorithm to get improved noise tolerance in the setting where the mixing weights are a priori known.…”

“…In this subsection we briefly remark that in the case where the mixing weights of D are known, we can combine Theorem 8.1 with the local convergence result of [KC19] to drive error down to even in settings where regression noise greatly exceeds .…”