Convergence of Adaptive, Randomized, Iterative Linear Solvers

Abstract

Deterministic and randomized, row-action and column-action linear solvers have become increasingly popular owing to their simplicity, low computational and memory complexities, and ease of composition with other techniques. Moreover, in order to achieve high-performance, such solvers must often be adapted to the given problem structure and to the hardware platform on which the problem will be solved. Unfortunately, determining whether such adapted solvers will converge to a solution has required equally unique analyses. As a result, adapted, reliable solvers are slow to be developed and deployed. In this work, we provide a general set of assumptions under which such adapted solvers are guaranteed to converge with probability one, and provide worst case rates of convergence. As a result, we can provide practitioners with guidance on how to design highly adapted, randomized or deterministic, row-action or column-action linear solvers that are also guaranteed to converge.

Randomized Block Adaptive Linear System Solvers

“…Thus, the supremum term is extremely important in finding better bounds on the rate of convergence. Moreover, the supremum term also underscores the importance of block methods over vector methods (see [26]) from a theoretical perspective. For vector methods, there are only two choices in the set Q i , and both produce the same value of Meany's constant.…”

“…In this work, we address the shortcomings of these previous works [12,27,26]. Specifically, we refine the three properties that we presented in [27,26] to be more precise for the vector case and to generalize to the block case, which allows our theory to cover the specific methods presented in [12] and many more (see section 4).…”

“…In this work, we address the shortcomings of these previous works [12,27,26]. Specifically, we refine the three properties that we presented in [27,26] to be more precise for the vector case and to generalize to the block case, which allows our theory to cover the specific methods presented in [12] and many more (see section 4). Thus, our theory enables the rigorous development and deployment of novel randomized solvers that take advantage of efficient block operations and that can be tailored to specific problems and computing environments.…”

“…Despite the desirable nature of a problem-blind compression of linear systems, randomized linear system solvers can suffer from poor performance if they fail to either account for problem structure or hardware considerations [26]. For instance, a randomized linear system solver that samples equations of a linear system that are not contiguously held in memory will incur expensive Input and Output (I/O) costs [26, §1].…”

“…In furtherance of this effort, we previously developed a novel framework for designing and analyzing the important class of randomized vector adaptive linear system solvers [27,26], which unified the analysis of such vector linear solvers as independent and identically distributed sampling randomized solvers [35,39,13,34]; greedy deterministic solvers [2,20]; randomized and deterministic cyclic sampling solvers [27]; greedy subsets followed by random selection solvers [5]; randomized subsets followed by greedy selection solvers [14]; and streaming solvers. More importantly, our framework and corresponding theory has displaced the need for the individual analysis of randomized vector adaptive solver, which enables the rapid prototyping and deployment of mathematically-sound randomized vector adaptive solvers that are highly-tailored to each unique problem and hardware environment.…”