Quantile-based Iterative Methods for Corrupted Systems of Linear Equations

Abstract

Often in applications ranging from medical imaging and sensor networks to error correction and data science (and beyond), one needs to solve large-scale linear systems in which a fraction of the measurements have been corrupted. We consider solving such large-scale systems of linear equations Ax = b that are inconsistent due to corruptions in the measurement vector b. We develop several variants of iterative methods that converge to the solution of the uncorrupted system of equations, even in the presence of large corruptions. These methods make use of a quantile of the absolute values of the residual vector in determining the iterate update. We present both theoretical and empirical results that demonstrate the promise of these iterative approaches. * The authors were partially supported by NSF CAREER #1348721 and NSF BIGDATA #1740325. We also acknowledge sponsorship by Capital FundManagement.

Quantile-Based Random Kaczmarz for corrupted linear systems of equations

“…The paper [19] also demonstrates, empirically, that the method works remarkably well and can handle both substantial amounts of error and real-life data.…”

“…We assume, throughout the paper, that all rows of A are normalized: a i ℓ 2 = 1. A fascinating approach was recently proposed by Haddock, Needell, Rebrova & Swartworth [19]: given an approximate solution x k , consider the set {| x k , a i − b i | : 1 ≤ i ≤ m}. This set measures, essentially, how 'wrong' each of the equations is.…”

“…The Random Kaczmarz method would now pick one of the equations uniformly at random. Haddock, Needell, Rebrova & Swartworth [19] propose to instead look at the q−th quantile of the set, these are the qm 'least incorrect' equations and then pick one at random and use that for a step of the Kaczmarz method (see also §4.1).…”

“…By looking at equations that are only violated 'a little', we are, hopefully, more likely to consider equations that are not corrupted (because corrupted equations are presumably violated by a lot); moreover, even if we were to pick an incorrect equation (one that was actually corrupted), selecting one with little overall error ensures that the error incurred in this step is not too large: the update may remove us from the correct solution but not by too much. Haddock, Needell, Rebrova & Swartworth [19] show that if A is a random matrix of a certain type, the method can recover the true solution with high likelihood as long as the support of the corruption β is sufficiently small.…”

“…Theorem (Haddock, Needell, Rebrova & Swartworth, [19]). For a certain class of random matrices A ∈ R m×n and m/n sufficiently large, if the support of the error is sufficiently small, β < min(c 1 q 2 , 1 − q), the q−quantile Random Kaczmarz method converges exponentially with likelihood at least 1 − c 2 exp(−c q m).…”