On the Convergence of Block Coordinate Descent Type Methods

Abstract

In this paper we study smooth convex programming problems where the decision variables vector is split into several blocks of variables. We analyze the block coordinate gradient projection method in which each iteration consists of performing a gradient projection step with respect to a certain block taken in a cyclic order. Global sublinear rate of convergence of this method is established and it is shown that it can be accelerated when the problem is unconstrained. In the unconstrained setting we also prove a sublinear rate of convergence result for the so-called alternating minimization method when the number of blocks is two. When the objective function is also assumed to be strongly convex, linear rate of convergence is established. Introduction.One of the first variable decomposition methods for solving general minimization problems is the so-called alternating minimization method [5,14], which is based on successive global minimization with respect to each component vector in a cyclic order. This fundamental method appears in the literature under various names such as the block-nonlinear Gauss-Seidel method or the block coordinate descent method (see, e.g., [4]). The convergence of the method was extensively studied in the literature under various assumptions. For example, Auslender studied in [1] the convergence of the method under a strong convexity assumption, but without assuming differentiability. In [4] Bertsekas showed that if the minimum with respect to each block of variables is unique, then any accumulation point of the sequence generated by the method is also a stationary point. Grippo and Sciandrone showed in [7] convergence results of the sequence generated by the method under different sets of assumptions such as strict quasi convexity with respect to each block. Luo and Tseng proved in [9] that under the assumptions of strong convexity with respect to each block, existence of a local error bound of the objective function, and proper separation of isocost surfaces, linear rate of convergence can be established.Another closely related method, which will be the main focus of this paper, is the block coordinate gradient projection (BCGP) method in which at each subiteration, the exact minimization with respect to a certain block of variables is replaced with an employment of a single step of the gradient projection method (a step toward the gradient followed by an orthogonal projection). This method has a clear advantage over alternating minimization when exact minimization with respect to each of the

Randomized Sparse Block Kaczmarz as Randomized Dual Block-Coordinate Descent

“…We consider the problem (2). A simple method for the solution of (2), called linearized Bregman method [22,6], is…”

“…For this particular choice (non-asymptotic) convergence rates were only recently derived in [2], although the convergence of the method was extensively studied in the literature under various assumptions [13,3]. Instead of using a deterministic cyclic order, randomized strategies were proposed in [14,12,16] for choosing a block to update at each iteration of the BCGD method.…”

“…In [2] the authors derive for the deterministic cyclic block coordinate gradient descent convergence rates which were not available before. They also compare the multiplicative constants in the convergence results above to the ones obtained for the deterministic cyclic block order.…”

“…The convergence of the method was shown in the framework of the split feasibilty problems [7,5]. We will follow a different approach and consider a block coordinate gradient descent method applied to the dual of (2). This dual problem is unconstrained and differentiable due to the -in particular -strict convexity of f .…”

Optimal power allocation for a full-duplex multicarrier decode-forward relay system with or without direct link

“…In the source phase, we compute the optimal source power allocation with a given relay power allocation; and in the relay phase, we compute the optimal relay power allocation for a given source power allocation. The two-phase iteration algorithm is a special case of block coordinate descent type methods , and is guaranteed to be locally convergent [38] .…”