An adaptive accelerated proximal gradient method and its homotopy continuation for sparse optimization

Abstract

We first propose an adaptive accelerated proximal gradient (APG) method for minimizing strongly convex composite functions with unknown convexity parameters. This method incorporates a restarting scheme to automatically estimate the strong convexity parameter and achieves a nearly optimal iteration complexity. Then we consider the ℓ 1 -regularized leastsquares (ℓ 1 -LS) problem in the high-dimensional setting. Although such an objective function is not strongly convex, it has restricted strong convexity over sparse vectors. We exploit this property by combining the adaptive APG method with a homotopy continuation scheme, which generates a sparse solution path towards optimality. This method obtains a global linear rate of convergence and its overall iteration complexity has a weaker dependency on the restricted condition number than previous work.

Adaptive restart of accelerated gradient methods under local quadratic growth condition

“…Then, as the rate of convergence depends on the match between the frequency and the quadratic error bound, we design a scheme to automatically adapt the frequency of restart from the observed decrease of the norm of the gradient mapping. The approach follows the lines of [Nes13,LX15,LY13]. We proved that, if our current estimate of the local error bound were correct, the norm of the gradient mapping would decrease at a prescribed rate.…”

“…The non-blowout property of accelerated schemes can be found in many papers, see for example [LX15,WCP17]. It will be repeatedly used in this paper to derive the linear convergence rate of restarted methods.…”

“…If (27) does not hold, then we know that µ > µ F (L, u 0 ). The idea was originally proposed by Nesterov in [Nes07] and later generalized in [LX15], where instead of restarting, they incorporate the estimate µ into the update of θ k . As a result, the complexity analysis only works for strongly convex objective function and seems not to hold under Assumption 2.…”

Backtracking Strategies for Accelerated Descent Methods with Smooth Composite Objectives

“…In [24] two heuristic restarting procedures based either on the evaluation of the composite functional or of a (generalised) gradient are studied. These two restarting approaches have become very popular since then and, more recently, some others have been proposed, for instance in [19] and [13]. Here, we follow [24] and apply the two function-and gradient-based restarting procedures to the GFISTA algorithm 2 with full backtracking to solve the Elastic Net problem above under the same choice of parameters as above.…”

A Novel TDOA-Based Localization Algorithm Using Sequential Quadratic Programming

“…This kind of algorithms requires initial values with sufficient accuracy to avoid local optimum. The other one converts nonlinear equations into linear forms via equation transformation with no error and produces approximate solutions using the least squares or weighted least squares (LS) method [4][5][6], which then develops into the linear-correction least square [7], total least squares [8,9], generalized total least squares [10], constrained total least squares [1], et al Among the algorithms mentioned above, the first kind that is based on initial models mostly depends on iteration for solution, with no guarantee for convergence and high complexity of computation. However, the result is relatively more accurate.…”