Optimal estimation of Gaussian mixtures via denoised method of moments

Abstract

The Method of Moments [Pea94] is one of the most widely used methods in statistics for parameter estimation, by means of solving the system of equations that match the population and estimated moments. However, in practice and especially for the important case of mixture models, one frequently needs to contend with the difficulties of non-existence or non-uniqueness of statistically meaningful solutions, as well as the high computational cost of solving large polynomial systems. Moreover, theoretical analyses of the method of moments are mainly confined to asymptotic normality style of results established under strong assumptions.This paper considers estimating a k-component Gaussian location mixture with a common (possibly unknown) variance parameter. To overcome the aforementioned theoretic and algorithmic hurdles, a crucial step is to denoise the moment estimates by projecting to the truncated moment space (via semidefinite programming) before solving the method of moments equations. Not only does this regularization ensures existence and uniqueness of solutions, it also yields fast solvers by means of Gauss quadrature. Furthermore, by proving new moment comparison theorems in the Wasserstein distance via polynomial interpolation and majorization techniques, we establish the statistical guarantees and adaptive optimality of the proposed procedure, as well as oracle inequality in misspecified models. These results can also be viewed as provable algorithms for Generalized Method of Moments [Han82] which involves non-convex optimization and lacks theoretical guarantees.

Asymptotics of smoothed Wasserstein distances

“…Our asymptotic results on the behavior of the χ 2 -divergence and relative entropy agree with several nonasymptotic bounds in the statistics literature for Gaussian mixtures [4,39]. To our knowledge, the asymptotic connection with smoothed Wasserstein distances is new.…”

Uniform Convergence Rates for Maximum Likelihood Estimation under Two-Component Gaussian Mixture Models

“…To the best of our knowledge, there are no existing works establishing minimax rates for parameter estimation in location-scale Gaussian mixture models with more than two components, except under the regime where the variances are presumed equal but unknown (Wu & Yang 2019). In future work, we intend to extend the analyses of this paper to Gaussian mixtures admitting more than two components.…”

“…In this context, the quantity K 0 is understood as the minimum number of well-separated components of the underlying mixture, with the case K 0 = 1 corresponding to the rate with no separation assumption. The minimax rate established by Heinrich & Kahn (2018) is achievable by a minimum-distance estimator, and by the Denoised Method of Moments (Wu & Yang 2019). A multivariate extension of the latter method was also shown to achieve the minimax rate of estimating a high-dimensional location-Gaussian mixture model (Doss et al 2020)-see also Wu & Zhou (2019) for the special case K = 2 of the minimax rate therein.…”

“…For mixture models failing to satisfy the strong identifiability condition, optimal rates for parameter estimation do not enjoy a unified treatment. For Gaussian mixture models with unknown means and common but unknown variances, Wu & Yang (2019) showed that the n − 1 4K rate is minimax optimal under no separation assumptions, and achievable by the Denoised Method of Moments. Feller et al (2019) shows this rate is also achievable by the MLE under two-component mixtures with equal variances.…”

The EM Algorithm is Adaptively-Optimal for Unbalanced Symmetric Gaussian Mixtures

“…1. The likelihood-based EM has method-of-moments alternatives [2,19,37] which may achieve the same error rates as the EM algorithm, perhaps at a higher computational cost. Specifically, for the balanced 2-GM model, the optimal error rate 4 is achieved by a spectral algorithm [38].…”