Sharp global convergence guarantees for iterative nonconvex optimization: A Gaussian process perspective

Chandrasekher, Kabir Aladin; Pananjady, Ashwin; Thrampoulidis, Christos

doi:10.48550/arxiv.2109.09859

Cited by 2 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, we establish entrywise guarantees on the EM algorithm-our results hold in the stronger ∞ -norm, not just the 2 -norm. Second, and in contrast to several results in the Gaussian case for mixtures of linear regressions (Kwon et al, 2019;Chandrasekher et al, 2021), we do not assume that each iterate of the EM algorithm takes in a fresh sample, i.e., our convergence guarantee is not based on sample-splitting 2 . Third, our result for the 2 -norm is sharp in that it also is optimal in terms of the constant factor, while most previous results for EM only achieve the optimal rate up to constant factors.…”

Section: Contributions and Organizationmentioning

confidence: 95%

“…As we see in the figure, the EM algorithm achieves a small error θ (T ) − θ * 2 2 with high probability 6 even if η is close to 1. We remark that there is a sizable literature on the success of iterative algorithms with random initialization for a variety of problems, such as phase retrieval , Gaussian mixtures (Dwivedi et al, 2020;Wu and Zhou, 2021), mixtures of logconcave distributions , and general regression models with Gaussian covariates (Chandrasekher et al, 2021). However, Gaussianity or continuous density is typically part of the assumption, and analyzing iterative algorithms beyond the Gaussian setting appears to be a generally more challenging problem.…”

Section: Failure Of Random Initializationmentioning

confidence: 99%

See 1 more Smart Citation

Sharp analysis of EM for learning mixtures of pairwise differences

Dhawan¹,

Cheng²,

Pananjady³

2023

Preprint

View full text Add to dashboard Cite

We consider a symmetric mixture of linear regressions with random samples from the pairwise comparison design, which can be seen as a noisy version of a type of Euclidean distance geometry problem. We analyze the expectation-maximization (EM) algorithm locally around the ground truth and establish that the sequence converges linearly, providing an ∞ -norm guarantee on the estimation error of the iterates. Furthermore, we show that the limit of the EM sequence achieves the sharp rate of estimation in the 2 -norm, matching the information-theoretically optimal constant. We also argue through simulation that convergence from a random initialization is much more delicate in this setting, and does not appear to occur in general. Our results show that the EM algorithm can exhibit several unique behaviors when the covariate distribution is suitably structured.

show abstract

Section: Contributions and Organizationmentioning

confidence: 95%

Section: Failure Of Random Initializationmentioning

confidence: 99%

Sharp analysis of EM for learning mixtures of pairwise differences

Dhawan¹,

Cheng²,

Pananjady³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…We analyze the cross-fitted AIPW estimator in the "proportional asymptotic regime", where the number of observations n and features p both diverge, with the ratio p/n converging to some constant κ > 0. This regime has attracted considerable recent attention in high-dimensional statistics [12, 13, 15-18, 23, 25, 29, 30, 38-40, 42, 44, 46, 55, 59, 61, 65, 82, 84, 94, 102, 104, 107, 109, 115, 117, 118], statistical machine learning and analysis of algorithms [31,36,60,67,68,71,72,74,79], econometrics [4,5,14,[26][27][28]51] etc, and shares roots with probability theory and statistical physics [78,120]. Asymptotic approximations derived under this regime demonstrate commendable performance even under moderate sample sizes (c.f.…”

mentioning

confidence: 99%

A New Central Limit Theorem for the Augmented IPW Estimator: Variance Inflation, Cross-Fit Covariance and Beyond

Jiang¹,

Mukherjee²,

Sen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Estimation of the average treatment effect (ATE) is a central problem in causal inference. In recent times, inference for the ATE in the presence of high-dimensional covariates has been extensively studied. Among the diverse approaches that have been proposed, augmented inverse propensity weighting (AIPW) with cross-fitting has emerged as a popular choice in practice. In this work, we study this cross-fit AIPW estimator under well-specified outcome regression and propensity score models in a high-dimensional regime where the number of features and samples are both large and comparable. Under assumptions on the covariate distribution, we establish a new CLT for the suitably scaled cross-fit AIPW that applies without any sparsity assumptions on the underlying high-dimensional parameters. Our CLT uncovers two crucial phenomena among others: (i) the AIPW exhibits a substantial variance inflation that can be precisely quantified in terms of the signal-to-noise ratio and other problem parameters, (ii) the asymptotic covariance between the precross-fit estimates is non-negligible even on the √ n scale. In fact, these crosscovariances turn out to be negative in our setting. These findings are strikingly different from their classical counterparts. On the technical front, our work utilizes a novel interplay between three distinct tools-approximate message passing theory, the theory of deterministic equivalents, and the leave-one-out approach. We believe our proof techniques should be useful for analyzing other two-stage estimators in this high-dimensional regime. Finally, we complement our theoretical results with simulations that demonstrate both the finite sample efficacy of our CLT and its robustness to our assumptions. * Author names sorted in alphabetical order.

show abstract

Sharp global convergence guarantees for iterative nonconvex optimization: A Gaussian process perspective

Cited by 2 publications

References 0 publications

Sharp analysis of EM for learning mixtures of pairwise differences

Sharp analysis of EM for learning mixtures of pairwise differences

A New Central Limit Theorem for the Augmented IPW Estimator: Variance Inflation, Cross-Fit Covariance and Beyond

Contact Info

Product

Resources

About