Randomized Kaczmarz with Averaging

2020

Preprint

Suppose A ∈ R n×n is invertible and we are looking for the solution of Ax = b. Given an initial guess x 1 ∈ R, we show that by reflecting through hyperplanes generated by the rows of A, we can generate an infinite sequence (x k ) ∞ k=1 such that all elements have the same distance to the solution, i.e.If the hyperplanes are chosen at random, averages over the sequence converge andThe bound does not depend on the dimension of the matrix. This introduces a purely geometric way of attacking the problem: are there fast ways of estimating the location of the center of a sphere from knowing many points on the sphere? Our convergence rate (coinciding with that of the Random Kaczmarz method) comes from averaging, can one do better?

“…, the Projection onto Convex Sets Method[2,6,7,11,36] and the Randomized Kaczmarz method[8,9,13,15,14,22,23,24,26,27,29,30,31,32,33,34,35,37,38,39,40,41,42].Strohmer & Vershynin …”

mentioning

confidence: 99%

Surrounding the solution of a Linear System of Equations from all sides

2020

Preprint

“…where A −1 2 is the operator norm of the inverse and A F is the Frobenius norm. This remarkable result has inspired a lot of subsequent work: we refer to [1,5,6,10,12,18,19,20,21,22,24,25,26,27,28,31,35,36,37] and references therein. It was recently shown in [33] that if v n ∈ R n denotes the singular vector corresponding to the smallest singular value of A, then…”

Section: Introductionmentioning

confidence: 99%

A Weighted Randomized Kaczmarz Method for Solving Linear Systems

2020

Preprint

The Kaczmarz method for solving a linear system Ax = b interprets such a system as a collection of equations a i , x = b i , where a i is the i−th row of A, then picks such an equation and corrects x k+1 = x k + λa i where λ is chosen so that the i−th equation is satisfied. Convergence rates are difficult to establish. Assuming the rows to be normalized, a i 2 = 1, Strohmer & Vershynin established that if the order of equations is chosen at random, E x k − x 2 converges exponentially. We prove that if the i−th row is selected with likelihood proportional to | a i , x k − b i | p , where 0 < p < ∞, then E x k −x 2 converges faster than the purely random method. As p → ∞, the method de-randomizes and explains, among other things, why the maximal correction method works well. We empirically observe that the method computes approximations of small singular vectors of A as a byproduct.

“…This is also known as the Algebraic Reconstruction Technique (ART) in computer tomography [10,14,15,26], the Projection onto Convex Sets Method [3,5,6,9,38] and the Randomized Kaczmarz method [2,7,8,11,13,12,20,21,22,24,25,27,28,29,30,31,33,36,39,40,41,42,43,44]. Strohmer & Vershynin [41] showed that…”

mentioning

confidence: 99%

On the Regularization Effect of Stochastic Gradient Descent applied to Least Squares

2020

Preprint

We study the behavior of stochastic gradient descent applied to Ax − b 2 2 → min for invertible A ∈ R n×n . We show that there is an explicit constant c A depending (mildly) on A such thatThis is a curious inequality: the last term has one more matrix applied to the residual u k − u than the remaining terms: if x k − x is mainly comprised of large singular vectors, stochastic gradient descent leads to a quick regularization. For symmetric matrices, this inequality has an extension to higher-order Sobolev spaces. This explains a (known) regularization phenomenon: an energy cascade from large singular values to small singular values smoothes.