Line search for averaged operator iteration

Abstract

Many popular first order algorithms for convex optimization, such as forward-backward splitting, Douglas-Rachford splitting, and the alternating direction method of multipliers (ADMM), can be formulated as averaged iteration of a nonexpansive mapping. In this paper we propose a line search for averaged iteration that preserves the theoretical convergence guarantee, while often accelerating practical convergence. We discuss several general cases in which the additional computational cost of the line search is modest compared to the savings obtained. IntroductionFirst-order algorithms such as forward-backward splitting, Douglas-Rachford splitting, and the alternating direction methods of multipliers (ADMM) are often used for large-scale convex optimization. While the theory tells us that these methods converge, practical convergence can be very slow for some problem instances. One effective method to reduce the number of iterations is to precondition the problem data. This approach has been extensively studied in the literature and has proven very successful in practice; see, e.g., [4,7,22,16,18,19] for a limited selection of such approaches.Another general approach to improving practical efficiency is to carry out a line search, i.e., to first compute a tentative next iterate and then to select the next iterate on the ray from the current iterate passing through the tentative iterate. Typical line searches are based on some readily computed quantity such as the function value or norm of the gradient or residual. A well designed line search preserves the theoretical convergence of the base method, while accelerating the practical convergence. Line search is widely used in gradient descent or Newton methods; see [6,24]. These line search methods cannot be applied to all first-order methods mentioned above, however, since in general there is no readily computed quantity that is decreasing. (The convergence proofs for these methods typically rely on quantities related to the distance to an optimal point, which cannot be evaluated while the algorithm is running.) In this paper we propose a general line search scheme that is applicable to most first-order convex optimization methods, including those mentioned above whose convergence proofs are not based on the decrease of an observable quantity.We exploit the fact that many first-order optimization algorithms can be viewed as averaged iterations of some nonexpansive operator, i.e., they can be written in the formwhereᾱ ∈ (0, 1) and S : R n → R n is nonexpansive, i.e., it satisfies Su−Sv 2 ≤ u − v 2 for all u, v. The superscript k denotes iteration number. The middle expression shows that the next point is a weighted average of the current point x k and Sx k . The expression on the righthand side of (1) shows that the iteration can be interpreted as a taking a step of lengthᾱ in the direction of the fixedpoint residual r k = Sx k − x k . Assuming a fixed-point exists, the iteration (1) converges to the set of fixed-points.In this paper we will show how steps som...

Line search for generalized alternating projections

“…However, by ensuring that the residual r(x k+1 ) is smaller than r(x iLS+1 ), we can guarantee that it will eventually decrease. This is proven for general line search schemes in [11] and we state it for the projected line search below.…”

“…A method for applying line search on algorithms based on iterating averaged operators was recently proposed in [11]. The method was shown to often improve practical convergence.…”

“…Cn+1 . The following iterations show that several candidate α k can be tested, without multiple evaluations of S 1 [11]:…”

Real‐time optimal energy management for a fuel cell/battery hybrid system

“…The convergence of proposed algorithm is proved in the following. As shown in [26], the algorithm is equivalent to…”

“…The convergence of proposed algorithm is proved in the following. As shown in , the algorithm is equivalent to where We define the primal residual and dual residual of the problem as …”