Solving separable nonlinear least squares problems using the QR factorization

An Efficient Algorithm for the Separable Nonlinear Least Squares Problem

“…We begin by presenting the relevant background results from [8]. Assume that A(y) and b(y) are twice Lipschitz continuously differentiable in a neighborhood of the least squares solution y * of (1).…”

“…In particular, we showed in [8] that our Gauss-Newton type method, and also the standard Gauss-Newton method [9][10][11], which omits the second derivative terms and is not quadratically convergent, are both invariant with respect to the specific basis matrix C(y) that is used at any particular point in the iteration. This makes it possible to substitute, at each point of the iteration, any easily computed local orthonormal basis for the nullspace of A T (y).…”

“…In the next section, we present the relevant results from [8] relating to our technique for solving (5) by our Gauss-Newton type method using QR factorization, and show how those results and methods can be modified to use LU factorization. The resulting algorithm and some numerical examples to illustrate the method are given in the following two sections.…”

“…In [8], we presented a quadratically convergent Gauss-Newton type iterative method, incorporating second derivative terms, to solve this problem, and provided a complete theoretical justification for our technique. In particular, we showed in [8] that our Gauss-Newton type method, and also the standard Gauss-Newton method [9][10][11], which omits the second derivative terms and is not quadratically convergent, are both invariant with respect to the specific basis matrix C(y) that is used at any particular point in the iteration.…”

“…Many instances of (1), such as those arising from the discretization of linear differential equations with only a few nonlinear parameters, involve N → ∞, while and n are fixed and small. In this case, the main computational cost in each step of our quadratically convergent Gauss-Newton type method of [8] is the QR factorization of A(y), costing approximately 4N 3 /3 flops [12] per iteration. In this note, we show how the relevant computations can instead be performed by using one LU factorization of A(y) per iteration, so that the computational cost of the computation is essentially halved to 2N 3 /3 flops [12].…”