Mixture models, robustness, and sum of squares proofs

Abstract

We use the Sum of Squares method to develop new efficient algorithms for learning wellseparated mixtures of Gaussians and robust mean estimation, both in high dimensions, that substantially improve upon the statistical guarantees achieved by previous efficient algorithms. Our contributions are:• Mixture models with separated means: We study mixtures of k distributions in d dimensions, where the means of every pair of distributions are separated by at least k ε . In the special case of spherical Gaussian mixtures, we give a (dk) O(1/ε 2 ) -time algorithm that learns the means assuming separation at least k ε , for any ε > 0. This is the first algorithm to improve on greedy ("single-linkage") and spectral clustering, breaking a long-standing barrier for efficient algorithms at separation k 1/4 .• Robust estimation: When an unknown (1−ε)-fraction of X 1 , . . . , X n are chosen from a sub-Gaussian distribution with mean µ but the remaining points are chosen adversarially, we give an algorithm recovering µ to error ε 1−1/t in time d O(t 2 ) , so long as sub-Gaussianness up to O(t) moments can be certified by a Sum of Squares proof. This is the first polynomial-time algorithm with guarantees approaching the information-theoretic limit for non-Gaussian distributions. Previous algorithms could not achieve error better than ε 1/2 .Both of these results are based on a unified technique. Inspired by recent algorithms of Diakonikolas et al. in robust statistics, we devise an SDP based on the Sum of Squares method for the following setting: given X 1 , . . . , X n ∈ R d for large d and n = poly(d) with the promise that a subset of X 1 , . . . , X n were sampled from a probability distribution with bounded moments, recover some information about that distribution.

High-Dimensional Robust Mean Estimation in Nearly-Linear Time

“…Can we obtain nearly-linear time robust algorithms for other inference tasks under sparsity assumptions [BDLS17] (e.g., for robust sparse mean estimation or robust sparse PCA)? Can we speed-up the convex programs obtained via the SoS hierarchy in this setting [HL18,KSS18]?…”

Polynomial Methods in Statistical Inference: Theory and Practice

“…However, in the present literature, most of the estimation procedures with finite sample guarantees are either clustering-based, or rely on separation conditions in the analysis (e.g. [156,157,158]). Bridging this conceptual divide is one of the main motivations of the present chapter.…”

Robustly Clustering a Mixture of Gaussians

“…Their technique, isotropic PCA, an affine-invariant version of PCA, is not robust (showing this is a bit more involved than for most spectral algorithms). So we turn for inspiration to the special case of spherical Gaussians for which robust algorithms have been recently discovered, with nearoptimal separation [16,11]. The key idea there is to express the identifiability of a Gaussian component in terms of a polynomial system, solve this polynomial system using a sum-of-squares semi-definite programming relaxation, and round the fractional solution obtained to a nearly correct clustering.…”

“…Over the past few years, there has been significant progress in computationally efficient robust estimation, starting with mean and covariance estimation for a large family of distributions [6,17]. Such robust estimation has also been discovered for various generative models including mixtures of well-separated spherical Gaussians (early work by Brubaker [3], and improved bounds more recently [6,11,16]), Independent Component Analysis [17,16] and linear regression. In spite of impressive progress, the core motivating problem of robustly estimating a mixture of (even two) Gaussians has remained unsolved.…”

“…The other directly relevant work is for mixtures of spherical Gaussians. Using the Sum-of-Squares convex programming hierarchy, it is possible to cluster a mixture of spherical Gaussians assuming pairwise mean separation that is close to the best possible in quasi-polynomial time; and in polytime for separation that is at least k ε standard deviations for any ε > 0 [16,11], improving on the polynomial-time k 1/4 -standarddeviations separation result of [21]. While the results of [21,4] are based on spectral methods and are not robust, the SoS-based results for spherical Gaussian mixtures are robust to adversarial noise in addition to having a weaker separation requirement.…”