Efficient estimation of regularization parameters via downsampling and the singular value expansion

Abstract

Abstract. The solution, x, of the linear system of equations Ax ≈ b arising from the discretization of an ill-posed integral equation with a square integrable kernel H(s, t) is considered. The Tikhonov regularized solution x(λ) is found as the minimizer of}. x(λ) depends on regularization parameter λ that trades off the data fidelity, and on the smoothing norm determined by L. Here we consider the case where L is diagonal and invertible, and employ the Galerkin method to provide the relationship between the singular value expansion and the singular value decomposition for square integrable kernels. The resulting approximation of the integral equation permits examination of the properties of the regularized solution x(λ) independent of the sample size of the data. We prove that estimation of the regularization parameter can be obtained by consistently down sampling the data and the system matrix, leading to solutions of coarse to fine grained resolution. Hence, the estimate of λ for a large problem may be found by downsampling to a smaller problem, or to a set of smaller problems, effectively moving the costly estimate of the regularization parameter to the coarse representation of the problem. Moreover, the full singular value decomposition for the fine scale system is replaced by a number of dominant terms which is determined from the coarse resolution system, again reducing the computational cost. Numerical results illustrate the theory and demonstrate the practicality of the approach for regularization parameter estimation using generalized cross validation, unbiased predictive risk estimation and the discrepancy principle applied for both the system of equations, and the augmented system of equations.

Adaptive Regularization Parameter Choice Rules for Large-Scale Problems

“…Future work includes the natural extension of the new class of algorithms to work with Krylov projection methods that are based on algorithms other than GKB (e.g., the Arnoldi algorithm or flexible Krylov methods; see [5]): while the computations involved in Algorithm 3 can be adapted to these situations, the theoretical analysis of the resulting strategies needs to be carefully rethought. Moreover, the new class of algorithms can be extended to handle Tikhonov-TSVD regularization, i.e., regularization methods that apply Tikhonov method to a TSVDprojected linear system (see, e.g., [27]): in these cases, one should replace the Krylov space K k appearing in Algorithm 3 by the space spanned by the first k right singular vectors of A; a careful theoretical analysis would be needed to prove convergence results. Also, other parameter choice strategies that can be expressed in the framework of bilevel optimization problems (e.g., the UPRE criterion, see [27] and the references therein) can be considered.…”

“…Moreover, the new class of algorithms can be extended to handle Tikhonov-TSVD regularization, i.e., regularization methods that apply Tikhonov method to a TSVDprojected linear system (see, e.g., [27]): in these cases, one should replace the Krylov space K k appearing in Algorithm 3 by the space spanned by the first k right singular vectors of A; a careful theoretical analysis would be needed to prove convergence results. Also, other parameter choice strategies that can be expressed in the framework of bilevel optimization problems (e.g., the UPRE criterion, see [27] and the references therein) can be considered. To conclude, the 44)).…”

“…Although the above relation is formally similar to (27), where L k = ∂ α P k and a zero finder is applied to ∂ α P (x(α)) = 0, applying (36) to the functionals (GCV), (QO), and (R) is not as straightforward as in Section 3.1, for a variety of reasons: firstly, ∂ α P (x(α)) may have multiple zeros (corresponding to local maxima or minima of P (x(α)); secondly, {∂ α P k (α)} k may not be nested upper or lower bounds for ∂ α P (x(α)); finally, some insight into how to choose k * is needed. Specific details are provided in the following subsections.…”

“…Given a sequence of functions {L k } k≥1 , the kth iteration of the modified Newton method (27) consists in performing only one (standard) Newton iteration on the kth function L k . Figure 1 gives a geometrical illustration of recursion (27). Remark 2.…”

“…Remark 3. Relation (30) is formally similar to (27). Notably, since G k (β) = f DP (β) for k ≥ p = min{n, m}, relation (30) reduces to (standard) Newton method when k ≥ p. However, this never happens in practice, because the bounds G k (β) are observed to quickly approach f DP (β) and the convergence of (standard) Newton method is quadratic; see also Section 4.…”