Learning regularization parameters of inverse problems via deep neural networks

Abstract

In this work, we describe a new approach that uses deep neural networks (DNN) to obtain regularization parameters for solving inverse problems. We consider a supervised learning approach, where a network is trained to approximate the mapping from observation data to regularization parameters. Once the network is trained, regularization parameters for newly obtained data are computed by efficient forward propagation of the DNN. We show that a wide variety of regularization functionals, forward models, and noise models may be considered. The network-obtained regularization parameters can be computed more efficiently and may even lead to more accurate solutions compared to existing regularization parameter selection methods. We emphasize that the key advantage of using DNNs for learning regularization parameters, compared to previous works on learning via bilevel optimization or empirical Bayes risk minimization, is greater generalizability. That is, rather than computing one set of parameters that is optimal with respect to one particular design objective, DNN-computed regularization parameters are tailored to the specific features or properties of the newly observed data. Thus, our approach may better handle cases where the observation is not a close representation of the training set. Furthermore, we avoid the need for expensive and challenging bilevel optimization methods as utilized in other existing training approaches. Numerical results demonstrate that trained DNNs can predict regularization parameters faster and better than existing methods, hence resulting in more accurate solutions to inverse problems.

The Variable Projected Augmented Lagrangian Method

“…Using the same computational setup as before, we consider varying sizes of the image in Figure 1, left panel. Note that in this setup we also slightly vary the regularization parameter µ ∈ [1,20] to obtain more realistic timings for "near optimal" regularization parameters. Additionally, in our comparison we include timings for a generalized Tikhonov approach of the form (1.3).…”

“…There is a substantive literature on determining the regularization parameter µ in (1.3), for which details are available in texts such as [39,40,3]. These range from techniques that do not need any information about the statistical distribution of the noise in the data, such as the L-curve that trades off between the data fit and the regularizer [40], the method of generalized cross-validation [32] and supervised learning techniques [53,1]. Some other approaches are statistically-based and require that an estimate of the variance of the noise in the data, assumed to be normal, is known.…”

“…Evaluation of the specific function relies on the solver for finding (1.3) but can generally be designed efficiently for 2 -regularization problems, see same references as above. In contrast, regularization parameter selection methods for 1 -regularization are generally more computationally demanding, because the methods require a full optimization solve of (1.2) for each choice of µ. Dependent on whether the underlying noise distribution is known, a few approaches have been proposed for defining an optimal choice of µ, including the use of the discrepancy principle (DP), computationally expensive cross-validation, and supervised learning techniques, [53,1].…”

“…In contrast, regularization parameter selection methods for 1 -regularization are generally more computationally demanding, because the methods require a full optimization solve of (1.2) for each choice of µ. Dependent on whether the underlying noise distribution is known, a few approaches have been proposed for defining an optimal choice of µ, including the use of the discrepancy principle (DP), computationally expensive cross-validation, and supervised learning techniques, [53,1]. In the statistical community, arguments based on the degrees of freedom (DF) in the obtained solutions are prevalent for finding µ, e.g., [67,46].…”

“…There are two main contributions of this work. (1) We introduce and analyze the VPAL method. We provide numerical evidence of the low computational complexity of the corresponding vpal algorithm.…”