Flexible Krylov Methods for $\ell_p$ Regularization

Abstract

In this paper we develop flexible Krylov methods for efficiently computing regularized solutions to large-scale linear inverse problems with an ℓ 2 fit-to-data term and an ℓ p penalization term, for p ≥ 1. First we approximate the p-norm penalization term as a sequence of 2-norm penalization terms using adaptive regularization matrices in an iterative reweighted norm fashion, and then we exploit flexible preconditioning techniques to efficiently incorporate the weight updates. To handle general (non-square) ℓ p -regularized least-squares problems, we introduce a flexible Golub-Kahan approach and exploit it within a Krylov-Tikhonov hybrid framework. The key benefits of our approach compared to existing optimization methods for ℓ p regularization are that efficient projection methods replace inner-outer schemes and that expensive regularization parameter selection techniques can be avoided. Theoretical insights are provided, and numerical results from image deblurring and tomographic reconstruction illustrate the benefits of this approach, compared to well-established methods. Furthermore, we show that our approach for p = 1 can be used to efficiently compute solutions that are sparse with respect to some transformations. 1where b ∈ R m is the observed data, A ∈ R m×n models the forward process, x true ∈ R n is the desired solution, and e ∈ R m represents noise or errors in the observation. Due to the ill-posedness of the underlying problem [18], regularization should be applied to recover a meaningful approximation of x true in (1). In this paper, we are interested in problems of the form minwhere · p for p ≥ 1 is the vectorial p-norm, λ > 0 is a regularization parameter, and Ψ ∈ R n×n is a nonsingular matrix. For p = 2 and Ψ = I, (2) is the standard Tikhonov regularization problem, and many efficient techniques, including hybrid iterative methods, have been proposed, see, e.g., [5,10,28,22]. However, optimization problems (2) for p = 2 can be significantly more challenging. For example, for p = 1, the so-called ℓ 1 -regularized problem suffers from non-differentiability at the origin; moreover, in some situations, one may wish to consider 0 < p < 1, which results in a nonconvex optimization problem, see, e.g., [20,24,25]. In this paper, we will focus on p ≥ 1, and henceforth we will refer to problem (2) with Ψ = I as an "ℓ p -regularized problem" and problem (2) with Ψ = I will be dubbed the "transformed ℓ p -regularized" problem. Typically the transformed ℓ p -regularized problem arises in cases where sparsity in some frequency domain (e.g., in a wavelet domain) is desired. Depending on the application, a sparsity transform may be included in both the fit-to-data and the regularization term. This was considered in [3] for image deblurring problems, where the resulting minimization problem was solved with an inner-outer iteration scheme.Most of the previously developed methods for ℓ p minimization utilize nonlinear optimization schemes or iteratively reweighted optimization schemes, which can get very expensive d...

Linearized Krylov subspace Bregman iteration with nonnegativity constraint

“…We now derive the proximal operator for J in (14) when W = I . Let W = {y; y = Wỹ,ỹ ≥ 0} (16) and let i W be the indicator function for the set W . Then…”

“…For anisotropic blurs, the choice of the Krylov subspace as in IRhybrid fgmres may not suitable. A more relevant choice of the Krylov subspace for these kinds of blurs is described in [16].…”

Krylov methods for inverse problems: Surveying classical, and introducing new, algorithmic approaches

“…The introduction of this new class of algorithms carries a number of open questions and possible extensions that may be addressed in future research. For instance, future work can include the natural extension of the new algorithms to work with Krylov projection methods that are based on decompositions other than GKB (eg, the Arnoldi decomposition or flexible Krylov methods; see [80]); also extensions to projection strategies that handle a generic regularization matrix L by computing joint decompositions of the coefficient matrix A and L (see, for instance, [63,81]) may be considered as an alternative to the strategy, employed within this paper, of transforming the Tikhonov problem into standard form and preconditioning the approximation subspace for the solution. Although the algorithmic details of the new approach can be easily adapted to these situations, the theoretical analysis of the resulting strategies needs to be carefully rethought.…”

Flexible GMRES for total variation regularization

“…Strategies to extend the TV-FGMRES method to incorporate additional penalization terms can be studied as well. Finally, ways of extending TV-FGMRES to handle non-square coefficient matrices can be devised, by exploiting the flexible Golub-Kahan bidiagonalization algorithm derived in [10].…”

“…, m, they form a basis for the vectorx L in (3.2). Therefore, at the mth step of TV-FGMRES,x L is approximated by the following vector 10) and e 1 ∈ R m+1 is the first canonical basis vector of R m+1 . Due to decomposition (3.9) and the properties of the matrices appearing therein, min…”