Stochastic algorithms with geometric step decay converge linearly on sharp functions

Davis, Damek; Drusvyatskiy, Dmitriy; Charisopoulos, Vasileios

doi:10.1007/s10107-023-02003-w

Math. Program.

2023

DOI: 10.1007/s10107-023-02003-w

|View full text |Cite

Stochastic algorithms with geometric step decay converge linearly on sharp functions

Damek Davis,

Dmitriy Drusvyatskiy,

Vasileios Charisopoulos

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2023

2024

Publication Types

Select...

Article3

Relationship

Self Cite0

Independent3

Authors

Journals

Cited by 3 publications

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Identifiability, the KL Property in Metric Spaces, and Subgradient Curves

Lewis,

Tian

2024

Found Comput Math

View full text Add to dashboard Cite

Identifiability, the KL Property in Metric Spaces, and Subgradient Curves

Lewis,

Tian

2024

Found Comput Math

View full text Add to dashboard Cite

Alternating minimization for generalized rank-1 matrix sensing: sharp predictions from a random initialization

Chandrasekher,

Lou,

Pananjady

2024

Information and Inference: A Journal of the IMA

View full text Add to dashboard Cite

We consider the problem of estimating the factors of a rank-$1$ matrix with i.i.d. Gaussian, rank-$1$ measurements that are nonlinearly transformed and corrupted by noise. Considering two prototypical choices for the nonlinearity, we study the convergence properties of a natural alternating update rule for this non-convex optimization problem starting from a random initialization. We show sharp convergence guarantees for a sample-split version of the algorithm by deriving a deterministic one-step recursion that is accurate even in high-dimensional problems. Notably, while the infinite-sample population update is uninformative and suggests exact recovery in a single step, the algorithm—and our deterministic one-step prediction—converges geometrically fast from a random initialization. Our sharp, non-asymptotic analysis also exposes several other fine-grained properties of this problem, including how the nonlinearity and noise level affect convergence behaviour. On a technical level, our results are enabled by showing that the empirical error recursion can be predicted by our deterministic one-step updates within fluctuations of the order $n^{-1/2}$ when each iteration is run with $n$ observations. Our technique leverages leave-one-out tools originating in the literature on high-dimensional $M$-estimation and provides an avenue for sharply analyzing complex iterative algorithms from a random initialization in other high-dimensional optimization problems with random data.

show abstract

MLR-SNet: Transferable LR Schedules for Heterogeneous Tasks

Shu

Zhu

Zhao

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.

Contact Info

customersupport@researchsolutions.com

10624 S. Eastern Ave., Ste. A-614

Henderson, NV 89052, USA

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Blog Terms and Conditions API Terms Privacy Policy Contact Cookie Preferences Do Not Sell or Share My Personal Information

Made with 💙 for researchers

Part of the Research Solutions Family.

Stochastic algorithms with geometric step decay converge linearly on sharp functions

Cited by 3 publications

References 32 publications

Identifiability, the KL Property in Metric Spaces, and Subgradient Curves

Identifiability, the KL Property in Metric Spaces, and Subgradient Curves

Alternating minimization for generalized rank-1 matrix sensing: sharp predictions from a random initialization

MLR-SNet: Transferable LR Schedules for Heterogeneous Tasks

Contact Info

Product

Resources

About