A Weighted Randomized Kaczmarz Method for Solving Linear Systems

Abstract

The Kaczmarz method for solving a linear system Ax = b interprets such a system as a collection of equations a i , x = b i , where a i is the i−th row of A, then picks such an equation and corrects x k+1 = x k + λa i where λ is chosen so that the i−th equation is satisfied. Convergence rates are difficult to establish. Assuming the rows to be normalized, a i 2 = 1, Strohmer & Vershynin established that if the order of equations is chosen at random, E x k − x 2 converges exponentially. We prove that if the i−th row is selected with likelihood proportional to | a i , x k − b i | p , where 0 < p < ∞, then E x k −x 2 converges faster than the purely random method. As p → ∞, the method de-randomizes and explains, among other things, why the maximal correction method works well. We empirically observe that the method computes approximations of small singular vectors of A as a byproduct.

Surrounding the solution of a Linear System of Equations from all sides

“…, the Projection onto Convex Sets Method[2,6,7,11,36] and the Randomized Kaczmarz method[8,9,13,15,14,22,23,24,26,27,29,30,31,32,33,34,35,37,38,39,40,41,42].Strohmer & Vershynin …”

Convergence of Adaptive, Randomized, Iterative Linear Solvers

“…To address this issue, building from our previous results [17], we specify a set of general conditions for such solvers under which we can guarantee convergence with probability one (w.p.1.). Moreover, we are also able to provide a worst case rate of convergence, which generalizes the theory for deterministic solvers [1,14] and complements the specialized mean-squared analyses for certain random solvers [23,10,6,2,7,22]. Thus, we are able to provide practitioners with a set of guiding principles to readily develop and deploy solvers that are highly adapted to their problem's structure and to their hardware platform, while also guaranteeing convergence.…”

“…In the latter case, we have not guaranteed convergence of the sequence, and, in order to do so, we must ensure the event of γ k → 1 as k → ∞ has probability zero. While we will present a general way of ensuring that this event holds with probability zero, we begin with a more special situation that includes the case where {w k } are standard basis elements [1,14,23,25,2,7,22].…”

“…Independent and Identically Distributed. We start by considering rowaction and column-action solvers in which {w k } are independent and identically distributed, which includes randomized Kaczmarz [23,22], randomized Coordinate Descent [25], and more general randomized vector sketching methods [6, §3.2 with B = I]. We now specify the behavior of ϕ and how our results can be applied.…”

“…As these examples show, it is easy to imagine a plethora of adaptive variants of row-action and column-action methods, both random and deterministic, that would take advantage of the unique problem structures and hardware considerations to readily increase the speed-to-solution. Unfortunately, heretofore, any such adaptive variants have required their own unique analyses (e.g., [1,14,23,10,6,2,7,22]). As a result, rigorous, adaptive iterative solvers have been difficult to develop and deploy.…”