Inner-Iteration Krylov Subspace Methods for Least Squares Problems

Morikuni, Keiichi; Hayami, Ken

doi:10.1137/110828472

Cited by 24 publications

(33 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, a QR factorization of A may be used, either a complete sparse QR factorization as offered by SuiteSparseQR [9] and qr mumps [5], or an incomplete QR factorization such as the multilevel incomplete QR (MIQR) factorization of Li and Saad [34]. Moreover, the results reported in [20,21] suggest that the BA-GMRES approach of Morikuni and Hayami [36,37] may offer a feasible alternative.…”

mentioning

confidence: 99%

On Using Cholesky-Based Factorizations and Regularization for Solving Rank-Deficient Sparse Linear Least-Squares Problems

Scott¹

2017

SIAM J. Sci. Comput.

View full text Add to dashboard Cite

Abstract. By examining the performance of modern parallel sparse direct solvers and exploiting our knowledge of the algorithms behind them, we perform numerical experiments to study how they can be used to efficiently solve rank-deficient sparse linear least-squares problems arising from practical applications. The Cholesky factorization of the normal equations breaks down when the least-squares problem is rank-deficient, while applying a symmetric indefinite solver to the augmented system can give an unacceptable level of fill in the factors. To try to resolve these difficulties, we consider a regularization procedure that modifies the diagonal of the unregularized matrix. This leads to matrices that are easier to factorize. We consider both the regularized normal equations and the regularized augmented system. We employ the computed factors of the regularized systems as preconditioners with an iterative solver to obtain the solution of the original (unregularized) problem. Furthermore, we look at using limited-memory incomplete Cholesky-based factorizations and how these can offer the potential to solve very large problems.Key words. least-squares problems, normal equations, augmented system, sparse matrices, direct methods, iterative methods, Cholesky factorizations, preconditioning, regularization AMS subject classifications. 65F05, 65F50 DOI. 10.1137/16M10653801. Introduction. In recent years, a number of methods have been proposed for preconditioning sparse linear least-squares problems; a brief overview with a comprehensive list of references is included in the introduction to the paper of Bru et al. [4]. The recent study of Gould and Scott [20,21] reviewed many of these methods (specifically those for which software has been made available) and then tested and compared their performance using a range of examples coming from practical applications. One of the outcomes of that study was some insight into which least-squares problems in the widely used sparse matrix collections CUTEst [19] and University of Florida [10] currently pose a real challenge for direct methods and/or iterative solvers. In particular, the study found that most of the available software packages were not reliable or efficient for rank-deficient least-squares problems (at least not when run with the recommended settings for the input parameters that were employed in the study). In this paper, we look further at such problems and focus on the effectiveness of both sparse direct solvers and iterative methods with incomplete factorization preconditioners. A key theme is the use of regularization (see, for example, [15,48,49]). We propose computing a factorization (either complete or incomplete) of a regularized problem and then using this as a preconditioner for an iterative solver to recover the solution of the original (unregularized) problem.The problem we are interested in is

show abstract

mentioning

confidence: 99%

On Using Cholesky-Based Factorizations and Regularization for Solving Rank-Deficient Sparse Linear Least-Squares Problems

Scott¹

2017

SIAM J. Sci. Comput.

View full text Add to dashboard Cite

show abstract

“…As shown, the major part in Algorithm 1 is to solve the LLS problem in step 6, which dominates the computational complexity of the dogleg method. To solve the LLS problem, we adopt the two typical iterative methods, CGLS [4,15] and BA-GMRES [14,20] methods. Thus, the preconditioned CGLS (PCGLS) method and BA-GMRES methods to solve the LLS problem (1.3) can be described as follows.…”

Section: Jacobian Free Trust Region Methodsmentioning

confidence: 99%

“…However, all these techniques do not work if we cannot form the full matrix or partial matrix of the Jacobian matrix explicitly in some Newton iterations. Thus, we will employ inner-iterations to construct the preconditioner implicitly as in [20]. The idea behind the inner-iteration is like this.…”

Section: Jacobian Free Trust Region Methodsmentioning

confidence: 99%

“…It only requires the objective function F(x) and subroutines to compute J (x k )v and J (x k ) T w. In other words, our implicit preconditioner construction can be used to determine preconditioners for all these existing Jacobian-free methods for nonlinear equations and NLS problems. Based on this preconditioner construction, we propose a Jacobian-free three-level trust region method, which consists of the dogleg/trust region framework, iterative solvers for LLS problems, such as BA-GMRES [14,20] and CGLS [4,15,25] and automatic differentiation (AD) [6,13] to compute J (x k )v and J (x k ) T w. Therefore, compared with other existing methods, our proposed method is suitable for general NLS problems and is problem independent.…”

Section: Introductionmentioning

confidence: 99%

“…Then, the popular LLS iterative solvers, CGLS [4,15,25] and BA-GMRES [20] methods, are applied to solve the LLS problem (1.3) or (1.4) for each outer level step without forming J (x) explicitly in the middle level. Finally, our proposed implicit preconditioner construction technique is set as the inner level to accelerate the convergence of the LLS iterative solvers in the middle level.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Jacobian-Free Implicit Inner-Iteration Preconditioner for Nonlinear Least Squares Problems

Zheng

Hayami

2016

J Sci Comput

Self Cite

View full text Add to dashboard Cite

Nonlinear least squares (NLS) problems arise in many applications. The common solvers require to compute and store the corresponding Jacobian matrix explicitly, which is too expensive for large problems. Recently, some Jacobian-free (or matrix free) methods were proposed, but most of these methods are not really Jacobian free since the full or partial Jacobian matrix still needs to be computed in some iteration steps. In this paper, we propose an effective real Jacobian free method especially for large NLS problems, which is realized by the novel combination of using automatic differentiation for J (x)v and J (x) T v along with the implicit iterative preconditioning ideas. Together, they yield a new and effective three-level iterative approach. In the outer level, the dogleg/trust region method is employed to solve the NLS problem. At each iteration of the dogleg method, we adopt the iterative linear least squares (LLS) solvers, CGLS or BA-GMRES method, to solve the LLS problem generated at each step of the dogleg method as the middle iteration. In order to accelerate the convergence of the iterative LLS solver, we propose an implicit inner iteration preconditioner based on the weighted Jacobi method. Compared to the existing Jacobian-free methods, our proposed three-level method need not compute any part of the Jacobian matrix explicitly in 123 J Sci Comput any iteration step. Furthermore, our method does not rely on the sparsity or structure pattern of the Jacobian, gradient or Hessian matrix. In other words, our method also works well for dense Jacobian matrices. Numerical experiments show the superiority of our proposed method.

show abstract

Implementation of interior-point methods for LP based on Krylov subspace iterative solvers with inner-iteration preconditioning

et al. 2019

Self Cite

View full text Add to dashboard Cite

We apply novel inner-iteration preconditioned Krylov subspace methods to the interior-point algorithm for linear programming (LP). Inner-iteration preconditioners recently proposed by Morikuni and Hayami enable us to overcome the severe ill-conditioning of linear equations solved in the final phase of interior-point iterations. The Krylov subspace methods do not suffer from rank-deficiency and therefore no preprocessing is necessary even if rows of the constraint matrix are not linearly independent. By means of these methods, a new interior-point recurrence is proposed in order to omit one matrix-vector product at each step. Extensive numerical experiments are conducted over diverse instances of 140 LP problems including the Netlib, QAPLIB, Mittelmann and Atomizer Basis Pursuit collections. The largest problem has 434,580 unknowns. It turns out that our implementation is more robust than the standard public domain solvers SeDuMi (Self-Dual Minimization), SDPT3 (Semidefinite Programming Toh-Todd-Tütüncü) and the LSMR iterative solver in PDCO (Primal-Dual Barrier Method for Convex Objectives) without increasing CPU time. The proposed interior-point method based on iterative solvers succeeds in solving a fairly large number of LP instances from benchmark libraries under the standard stopping criteria. The work also presents a fairly extensive benchmark test for several renowned solvers including direct and iterative solvers.

show abstract

Inner-Iteration Krylov Subspace Methods for Least Squares Problems

Cited by 24 publications

References 39 publications

On Using Cholesky-Based Factorizations and Regularization for Solving Rank-Deficient Sparse Linear Least-Squares Problems

On Using Cholesky-Based Factorizations and Regularization for Solving Rank-Deficient Sparse Linear Least-Squares Problems

Jacobian-Free Implicit Inner-Iteration Preconditioner for Nonlinear Least Squares Problems

Implementation of interior-point methods for LP based on Krylov subspace iterative solvers with inner-iteration preconditioning

Contact Info

Product

Resources

About