Clustering a Mixture of Gaussians with Unknown Covariance

Davis, Damek; Díaz, Mateo; Wang, Kaizheng

doi:10.48550/arxiv.2110.01602

Cited by 5 publications

(11 citation statements)

References 89 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, [DDW21] showed that the MLE estimator for the missing labels corresponds to a Max-Cut problem, which recovers the solution when n = Ω(d). Moreover, the authors argued that while SNR drives the inherent statistical difficulty of the estimation problem, a relaxed quantity S = v 2 / Σ presumably controls the computational difficulty.…”

Section: Relation To Prior Workmentioning

confidence: 74%

“…In particular, this improves the state-of-the-art for this task, and provably beats all low-degree algorithms, when ρ = ω(1/ √ d). The corollary is based on the equivalence (up to rescaling) between Model 3.5 and some appropriate sub-case of Model 3.2 [DDW21]. Proof.…”

Section: Implications Of the Success Of Lllmentioning

confidence: 99%

“…Several previous works [BV08, MV10, BDJ + 20, CMZ19, FPB17] introduce algorithms that either require larger sample complexity n = Ω(d 2 ), or have non-optimal error rates, for instance based on k-means relaxations [Roy17, MVW17, GV19, LLL + 20]. Leveraging existing SoS lower bounds on the associated Max-Cut problem (see Section 1.3), [DDW21] suggest a statisticalto-computational gap for exact recovery in the binary Gaussian mixture. Our main result (Theorem 1.1) implies that this problem can be solved via the LLL basis reduction method in polynomial time whenever n ≥ d + 1.…”

Section: Relation To Prior Workmentioning

confidence: 99%

“…Prior to our work, the best known poly-time algorithm required n ≫ d 2 samples 1 [MW21]. Furthermore, this was believed to be unimprovable due to lower bounds against SoS algorithms and low-degree polynomials [MRX20, GJJ + 20, MW21,DDW21]. Nevertheless, we give a poly-time algorithm under the much weaker assumption n ≥ d + 1.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Lattice-Based Methods Surpass Sum-of-Squares in Clustering

Zadik¹,

Song²,

Wein³

et al. 2021

Preprint

View full text Add to dashboard Cite

Clustering is a fundamental primitive in unsupervised learning which gives rise to a rich class of computationally-challenging inference tasks. In this work, we focus on the canonical task of clustering d-dimensional Gaussian mixtures with unknown (and possibly degenerate) covariance. Recent works (Ghosh et al. '20; Mao, Wein '21; Davis, Diaz, Wang '21) have established lower bounds against the class of low-degree polynomial methods and the sum-of-squares (SoS) hierarchy for recovering certain hidden structures planted in Gaussian clustering instances. Prior work on many similar inference tasks portends that such lower bounds strongly suggest the presence of an inherent statistical-to-computational gap for clustering, that is, a parameter regime where the clustering task is statistically possible but no polynomial-time algorithm succeeds.One special case of the clustering task we consider is equivalent to the problem of finding a planted hypercube vector in an otherwise random subspace. We show that, perhaps surprisingly, this particular clustering model does not exhibit a statistical-to-computational gap, even though the aforementioned low-degree and SoS lower bounds continue to apply in this case. To achieve this, we give a polynomial-time algorithm based on the Lenstra-Lenstra-Lovasz lattice basis reduction method which achieves the statistically-optimal sample complexity of d + 1 samples. This result extends the class of problems whose conjectured statistical-to-computational gaps can be "closed" by "brittle" polynomial-time algorithms, highlighting the crucial but subtle role of noise in the onset of statistical-to-computational gaps.

show abstract

Section: Relation To Prior Workmentioning

confidence: 74%

Section: Implications Of the Success Of Lllmentioning

confidence: 99%

Section: Relation To Prior Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Lattice-Based Methods Surpass Sum-of-Squares in Clustering

Zadik¹,

Song²,

Wein³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Similarly, the assumption of isotropic noise is relaxable as long as the covariance tensor is known. The case of unknown covariance is much more challenging (Davis et al, 2021;Bakshi et al, 2020;Belkin and Sinha, 2010;Cai et al, 2019;Ge et al, 2015;Moitra and Valiant, 2010) even in the vector case and is beyond the scope of the current paper.…”

Section: Introductionmentioning

confidence: 98%

Optimal Estimation and Computational Limit of Low-rank Gaussian Mixtures

Lyu¹,

Dong²

2022

Preprint

View full text Add to dashboard Cite

Structural matrix-variate observations routinely arise in diverse fields such as multi-layer network analysis and brain image clustering. While data of this type have been extensively investigated with fruitful outcomes being delivered, the fundamental questions like its statistical optimality and computational limit are largely under-explored. In this paper, we propose a lowrank Gaussian mixture model (LrMM) assuming each matrix-valued observation has a planted low-rank structure. Minimax lower bounds for estimating the underlying low-rank matrix are established allowing a whole range of sample sizes and signal strength. Under a minimal condition on signal strength, referred to as the information-theoretical limit or statistical limit, we prove the minimax optimality of a maximum likelihood estimator which, in general, is computationally infeasible. If the signal is stronger than a certain threshold, called the computational limit, we design a computationally fast estimator based on spectral aggregation and demonstrate its minimax optimality. Moreover, when the signal strength is smaller than the computational limit, we provide evidences based on the low-degree likelihood ratio framework to claim that no polynomial-time algorithm can consistently recover the underlying low-rank matrix. Our results reveal multiple phase transitions in the minimax error rates and the statistical-to-computational gap. Numerical experiments confirm our theoretical findings. We further showcase the merit of our spectral aggregation method on the worldwide food trading dataset.

show abstract

Minimax Supervised Clustering in the Anisotropic Gaussian Mixture Model: A new take on Robust Interpolation

Minsker¹,

Ndaoud²,

Shen³

2021

Preprint

View full text Add to dashboard Cite

We study the supervised clustering problem under the twocomponent anisotropic Gaussian mixture model in high dimensions and in the non-asymptotic setting. We first derive a lower and a matching upper bound for the minimax risk of clustering in this framework. We also show that in the high-dimensional regime, the linear discriminant analysis (LDA) classifier turns out to be sub-optimal in the minimax sense. Next, we characterize precisely the risk of 2 -regularized supervised least squares classifiers. We deduce the fact that the interpolating solution may outperform the regularized classifier, under mild assumptions on the covariance structure of the noise. Our analysis also shows that interpolation can be robust to corruption in the covariance of the noise when the signal is aligned with the "clean" part of the covariance, for the properly defined notion of alignment. To the best of our knowledge, this peculiar phenomenon has not yet been investigated in the rapidly growing literature related to interpolation. We conclude that interpolation is not only benign but can also be optimal, and in some cases robust.

show abstract

Clustering a Mixture of Gaussians with Unknown Covariance

Cited by 5 publications

References 89 publications

Lattice-Based Methods Surpass Sum-of-Squares in Clustering

Lattice-Based Methods Surpass Sum-of-Squares in Clustering

Optimal Estimation and Computational Limit of Low-rank Gaussian Mixtures

Minimax Supervised Clustering in the Anisotropic Gaussian Mixture Model: A new take on Robust Interpolation

Contact Info

Product

Resources

About