Randomized block Kaczmarz method with projection for solving least squares

Abstract

The Kaczmarz method is an iterative method for solving overcomplete linear systems of equations Ax = b. The randomized version of the Kaczmarz method put forth by Strohmer and Vershynin iteratively projects onto a randomly chosen solution space given by a single row of the matrix A and converges linearly 1 in expectation to the solution of a consistent system. In this paper we analyze two block versions of the method each with a randomized projection, designed to converge in expectation to the least squares solution, often faster than the standard variants. Our approach utilizes both a row and column-paving of the matrix A to guarantee linear convergence when the matrix has consistent row norms (called nearly standardized), and a single column-paving when the row norms are unbounded. The proposed methods are an extension of the block Kaczmarz method analyzed by Needell and Tropp and the Randomized Extended Kaczmarz method of Zouzias and Freris. The contribution is thus two-fold; unlike the standard Kaczmarz method, our results demonstrate convergence to the least squares solution of inconsistent systems (both methods in the nearly standardized case and the second method in other cases). By using appropriate blocks of the matrix this convergence can be significantly accelerated, as is demonstrated by numerical experiments.

Convergence analysis for Kaczmarz-type methods in a Hilbert space framework

“…The analysis of block-iterative methods requires different splittings, based either on row or on column partitionings of A. We mention one such splitting that is related to the results about block-Kaczmarz solvers in [4,5]. Consider the same V a and variational problem (2) as in Example 2.…”

“…One case, where the estimation of these constants is relatively easy but does not lead to a small κ T , is worth mentioning. Referring to results concerning the optimal paving of operators on Hilbert spaces, the authors of [4,5] consider special row partitions, characterized by the property that there exist positive constants 0 < α < β < ∞ such that…”

“…Therefore, row partitions T satisfying (6) with β/α close to 1, as discussed in [4,5], do not have any preconditioning effect on solving Ax = b but guarantee fast solvability of the subproblems in block-Kaczmarz iterations considered in Section 3.2.…”

“…This paper is a reaction to a number of recent publications [1][2][3][4][5] on randomized versions of the Kaczmarz method triggered by Strohmer and Vershynin [6], and should be viewed as an addendum to [7,8]. The latter two papers are devoted to the theory of so-called Schwarz iterative (or subspace correction) methods for solving elliptic variational problems in Hilbert spaces.…”

A Sampling Kaczmarz--Motzkin Algorithm for Linear Feasibility

“…Zouzias and Freris [ZF12] analyzed an extended randomized Kaczmarz method which incorporates an additional projection step to reduce the size of the residual. This was extended to the block case in [NZZ15]. The relation of these approaches to coordinate descent and gradient descent methods has also been recently studied, see e.g.…”