Majorization-Minimization-Based Levenberg--Marquardt Method for Constrained Nonlinear Least Squares

Marumo, Naoki; Okuno, Takayuki

doi:10.48550/arxiv.2004.08259

Cited by 1 publication

(6 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…by solving subproblem (1.2) exactly. Solving the subproblem inexactly with first-order methods yields LM methods (Bergou et al, 2020;Marumo et al, 2020) that enjoy the iteration and oracle complexity bounds, which are the same order as (2.1).…”

Section: Iteration and Oracle Complexitymentioning

confidence: 99%

“…Pioneered by Yamashita and Fukushima (2001), many studies have shown local quadratic convergence under these assumptions or essentially equivalent assumptions (Behling and Fischer, 2012;Bellavia and Morini, 2015;Bellavia et al, 2010;Dan et al, 2002;Facchinei et al, 2013;Fan and Pan, 2006;Fan and Yuan, 2005;Fischer et al, 2010;Kanzow et al, 2004). Under the same assumptions, Marumo et al (2020) proposed an LM method that achieves both an iteration complexity bound and quadratic convergence. 4 Bergou et al (2020) showed linear convergence with small ∆ ∞ and the quadratic growth.…”

Section: Local Convergencementioning

confidence: 99%

“…Assumptions 1(e) and 1(f) impose the Lipschitz continuity of ∇h and ∇c on R n and dom g. These assumptions can be restricted to the sublevel sets as in the existing analysis (Marumo et al, 2020); we can also show the complexity bounds and local convergence shown in Sections 4 and 5 by assuming…”

Section: Assumptions On Problem (11)mentioning

confidence: 99%

“…We should note that Eq. (3.5) is inspired by the existing choice of µ = ρ c(x k ) for the case where g is an indicator function and Marumo et al, 2020). Algorithm 1 employs backtracking to choose the parameter ρ and ensures that the objective function value decreases sufficiently; for (x k ) k∈N and (µ k ) k∈N obtained by Algorithm 1, we have…”

Section: Algorithmmentioning

confidence: 99%

“…To evaluate the global convergence rate of LM methods, several studies (Bergou et al, 2020;Drusvyatskiy and Lewis, 2018;Marumo et al, 2020;Ueda and Yamashita, 2010;Zhao and Fan, 2016) have derived upper bounds on iteration complexity, which is the number of iterations required to find an ε-stationary point. The iteration complexity makes sense as a measure for the whole running cost if we explicitly compute c(x k ) and ∇c(x k ) before solving subproblem (1.2) and their computational costs are dominant; then, the total cost of LM is proportional to the iteration complexity.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Accelerated-gradient-based generalized Levenberg--Marquardt method with oracle complexity bound and local quadratic convergence

Marumo¹,

Okuno²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Many machine learning tasks are formulated as a nonconvex optimization problem of minimizing the sum of a convex function and a composite function. The generalized Levenberg-Marquardt (LM) method, also known as the prox-linear method, has been developed for such problems. The method iteratively solves strongly convex subproblems with a damping term.This study proposes a new generalized LM method for solving the problem with a smooth composite function. The method enjoys three theoretical guarantees: iteration complexity bound, oracle complexity bound, and local convergence under a Hölderian growth condition. The local convergence results include local quadratic convergence under the quadratic growth condition; this is the first to extend a classical result for least-squares problems to a general smooth composite function. In addition, this is the first LM method with both an oracle complexity bound and local quadratic convergence under standard assumptions. These results are achieved by carefully controlling the damping parameter and solving the subproblems by the accelerated proximal gradient method equipped with a particular termination condition. Experimental results show that the proposed method performs well in practice for several instances, including classification with a neural network and collaborative filtering with nonnegative matrix factorization.

show abstract

Section: Iteration and Oracle Complexitymentioning

confidence: 99%