Component-Averaged Row Projections: A Robust, Block-Parallel Scheme for Sparse Linear Systems

Abstract

Abstract.A new method for the parallel solution of large sparse linear systems is introduced. It proceeds by dividing the equations into blocks and operating in block-parallel iterative mode; i.e., all the blocks are processed in parallel, and the partial results are "merged" to form the next iterate. The new scheme performs Kaczmarz row projections within the blocks and merges the results by certain component-averaging operations-hence it is called component-averaged row projections, or CARP. The system matrix can be general, nonsymmetric, and ill-conditioned, and the division into blocks is unrestricted. For partial differential equations (PDEs), if the blocks are domain-based, then only variables at the boundaries between domains are averaged, thereby minimizing data transfer between processors. CARP is very robust; its application to test cases of linear systems derived from PDEs shows that it converges in difficult cases where state-of-the-art methods fail. It is also very memory efficient and exhibits an almost linear speedup ratio, with efficiency greater than unity in some cases. A formal proof of convergence is presented: It is shown that the component-averaging operations are equivalent to row projections in a certain superspace, so the convergence properties of CARP are identical to those of Kaczmarz's algorithm in the superspace. CARP and its convergence proof also apply to the consistent convex feasibility problem. 1. Introduction. Iterative methods for solving large sparse linear systems of equations are advantageous over the classical direct solvers, especially for huge systems. Methods of parallelizing iterative algorithms [20,29] divide the equations into blocks and usually fall into one of two modes of operation. In the block-sequential (also called block-iterative) mode, the blocks are processed sequentially, but the computations on each block's equations are done in parallel. Examples of block-sequential methods are found in [1,6,10]. In the second mode of operation, sometimes referred to as block-parallel, the blocks are assigned to different processors to be processed in parallel, and the results from all the blocks are then combined in some manner to produce the next iterate. Typical examples of block-parallel schemes are parallel versions of , RGMRES [30], conjugate gradient (CG), and CG-squared (CGS) [31]; other examples include those in [2,4,9,12,13].Kaczmarz's row projection method (KACZ) [23] was one of the first iterative methods used for large nonsymmetric systems. Its main advantages are robustness, guaranteed convergence on consistent systems, and cyclic convergence on inconsistent systems [8,16,32]. KACZ was also independently discovered in the context of image reconstruction from projections, where it is called ART (algebraic reconstruction technique) [21]. Kaczmarz's algorithm, by its nature and mathematical definition, is inherently sequential since, at each iterative step, the current iterate is projected

Bounds on the largest singular value of a matrix and the convergence of simultaneous and block‐iterative algorithms for sparse linear systems

“…For w i = 1, this is the CARP1 algorithm of [13] (see also [11,8,9]). The simultaneous DROP algorithm of [7] requires only that the weights w i be positive, but dividing each w i by their maximum, max i {w i }, while multiplying each λ k by the same maximum,…”

“…Convergence of CAV then follows, as does convergence of several other methods, including the ART, Landweber's method, the SART [1], the block-iterative CAV (BICAV) [9], the CARP1 method of Gordon and Gordon [13], and a block-iterative variant of CARP1 obtained from the DROP method of Censor et al [7]. Convergence of most of these methods was also established in [15], using a unifying framework of a block-iterative Landweber algorithm, but without deriving upper bounds for the largest eigenvalue of a general A † A.…”

Multicore Performance of Block Algebraic Iterative Reconstruction Methods

“…Gordon and Gordon [23] introduced an alternative way to combined the intermediate vectors x k,1 , . .…”

3D Frequency-Domain Seismic Inversion with Controlled Sloppiness

“…The latter does not make these approaches very attractive for inversion since the medium parameters will change from one iteration to the next, possibly requiring the tuning parameters to be changed as well. Instead, we use CARP-CG [14,15], a generic, iterative solution technique for sparse linear systems based on the CGMN method [5]. Convergence of this method is guaranteed, making it an attractive solver for inversion purposes.…”

“…Although these expressions are convenient for analysis and testing, a more efficient implementation computes only the action of these matrices using Algorithm 1 as Qu + Rs = DKSWP(A, u, s, γ) [14].…”