A trust region algorithm with a worst-case iteration complexity of $$\mathcal{O}(\epsilon ^{-3/2})$$ O ( ϵ - 3 / 2 ) for nonconvex optimization

Curtis, Frank E.; Robinson, Daniel N.; Samadi, Mohammadreza

doi:10.1007/s10107-016-1026-2

Cited by 122 publications

(156 citation statements)

References 11 publications

Supporting

Mentioning

154

Contrasting

Order By: Relevance

“…Both methods are capable of finding an ε-second-order stationary point (cf. (155)) in O(ε −1.5 ) iterations [185,189], where each iteration consists of one gradient and one Hessian computations.…”

Section: Hessian-based Algorithmsmentioning

confidence: 99%

Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview

Chi

Chen

2019

IEEE Trans. Signal Process.

336

284

View full text Add to dashboard Cite

Substantial progress has been made recently on developing provably accurate and efficient algorithms for low-rank matrix factorization via nonconvex optimization. While conventional wisdom often takes a dim view of nonconvex optimization algorithms due to their susceptibility to spurious local minima, simple iterative methods such as gradient descent have been remarkably successful in practice. The theoretical footings, however, had been largely lacking until recently.In this tutorial-style overview, we highlight the important role of statistical models in enabling efficient nonconvex optimization with performance guarantees. We review two contrasting approaches: (1) two-stage algorithms, which consist of a tailored initialization step followed by successive refinement; and (2) global landscape analysis and initialization-free algorithms. Several canonical matrix factorization problems are discussed, including but not limited to matrix sensing, phase retrieval, matrix completion, blind deconvolution, robust principal component analysis, phase synchronization, and joint alignment. Special care is taken to illustrate the key technical insights underlying their analyses. This article serves as a testament that the integrated consideration of optimization and statistics leads to fruitful research findings.

show abstract

Section: Hessian-based Algorithmsmentioning

confidence: 99%

Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview

Chi

Chen

2019

IEEE Trans. Signal Process.

336

284

View full text Add to dashboard Cite

show abstract

“…A trust region method with an O(ǫ −3/2 g ) complexity for achieving approximate first-order stationarity was proposed and analyzed in [3]. This method can be seen, along with that in [1], as a special case of the general framework in [4] for achieving this order complexity.…”

Section: A Strategy With a Fixed Trust Region Radiusmentioning

confidence: 99%

Concise complexity analyses for trust region methods

2018

Self Cite

View full text Add to dashboard Cite

Concise complexity analyses are presented for simple trust region algorithms for solving unconstrained optimization problems. In contrast to a traditional trust region algorithm, the algorithms considered in this paper require certain control over the choice of trust region radius after any successful iteration. The analyses highlight the essential algorithm components required to obtain certain complexity bounds. In addition, a new update strategy for the trust region radius is proposed that offers a second-order complexity bound.

show abstract

“…Crucially, examples are known for which such order estimates are tight both for trust-region and regularization methods [5]. Of late, more sophisticated trust region methods and quadratic regularization ones have been proposed that echo the order of the ARC estimates [9,14,2]. At the same time, other methods [10,12] have been shown to mirror the TR-like evaluation estimate in a more general or simplified way, respectively.…”

Section: Introductionmentioning

confidence: 99%

A concise second-order complexity analysis for unconstrained optimization using high-order regularized models

Cartis

Gould

Toint

2019

Optimization Methods and Software

View full text Add to dashboard Cite

The unconstrained minimization of a sufficiently smooth objective function f (x) is considered, for which derivatives up to order p, p ≥ 2, are assumed to be available. An adaptive regularization algorithm is proposed that uses Taylor models of the objective of order p and that is guaranteed to find a first-and second-order critical point in atfunction and derivatives evaluations, where ǫ 1 and ǫ 2 > 0 are prescribed first-and second-order optimality tolerances. Our approach extends the method in Birgin et al. (2016) to finding second-order critical points, and establishes the novel complexity bound for second-order criticality under identical problem assumptions as for first-order, namely, that the p-th derivative tensor is Lipschitz continuous and that f (x) is bounded from below. The evaluation-complexity bound for second-order criticality improves on all such known existing results.

show abstract

A trust region algorithm with a worst-case iteration complexity of $$\mathcal{O}(\epsilon ^{-3/2})$$ O ( ϵ - 3 / 2 ) for nonconvex optimization

Cited by 122 publications

References 11 publications

Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview

Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview

Concise complexity analyses for trust region methods

A concise second-order complexity analysis for unconstrained optimization using high-order regularized models

Contact Info

Product

Resources

About