Globally Convergent Optimization Algorithms on Riemannian Manifolds: Uniform Framework for Unconstrained and Constrained Optimization

Parameter estimation by nonlinear least squares minimization is a common problem that has an elegant geometric interpretation: the possible parameter values of a model induce a manifold within the space of data predictions. The minimization problem is then to find the point on the manifold closest to the experimental data. We show that the model manifolds of a large class of models, known as sloppy models, have many universal features; they are characterized by a geometric series of widths, extrinsic curvatures, and parameter-effects curvatures, which we describe as a hyper-ribbon. A number of common difficulties in optimizing least squares problems are due to this common geometric structure. First, algorithms tend to run into the boundaries of the model manifold, causing parameters to diverge or become unphysical before they have been optimized. We introduce the model graph as an extension of the model manifold to remedy this problem. We argue that appropriate priors can remove the boundaries and further improve the convergence rates. We show that typical fits will have many evaporated parameters unless the data are very accurately known. Second, 'bare' model parameters are usually ill-suited to describing model behavior; cost contours in parameter space tend to form hierarchies of plateaus and long narrow canyons. Geometrically, we understand this inconvenient parameterization as an extremely skewed coordinate basis and show that it induces a large parameter-effects curvature on the manifold. By constructing alternative coordinates based on geodesic motion, we show that these long narrow canyons are transformed in many cases into a single quadratic, isotropic basin. We interpret the modified Gauss-Newton and Levenberg-Marquardt fitting algorithms as an Euler approximation to geodesic motion in these natural coordinates on the model manifold and the model graph respectively. By adding a geodesic acceleration adjustment to these algorithms, we alleviate the difficulties from parameter-effects curvature, improving both efficiency and success rates at finding good fits.

show abstract

“…We have here offered an explanation of this effect based on the assumption that the deviation from flatness is given by Eq. (32).…”

Section: Curvature In Sloppy Modelsmentioning

confidence: 99%

Geometry of nonlinear least squares with applications to sloppy models and optimization

2011

View full text Add to dashboard Cite

show abstract

“…Finally, the presence of a trust-region gives an additional guideline to stop the inner iteration early, hence reducing the computational cost, while preserving the fast local convergence of the exact scheme. Line-search techniques have been considered on Riemannian manifolds by Udrişte [Udr94] and Yang [Yan06].…”

Section: Introductionmentioning

confidence: 99%

Trust-Region Methods on Riemannian Manifolds

2006

View full text Add to dashboard Cite

show abstract

“…A similar definition was proposed in [Yan07] for the particular case where the retraction is the exponential mapping. When M = R n with its canonical Euclidean structure, the definition reduces to the classical situation described, e.g., in [Ber95].…”

Section: Accelerated Line-search Methodsmentioning

confidence: 94%

Accelerated Line-search and Trust-region Methods

Absil¹,

Gallivan²

2009

SIAM J. Numer. Anal.

View full text Add to dashboard Cite

In numerical optimization, line-search and trust-region methods are two important classes of descent schemes, with well-understood global convergence properties. Here we consider "accelerated" versions of these methods, where the conventional iterate is allowed to be replaced by any point that produces at least as much decrease in the cost function as a fixed fraction of the decrease produced by the conventional iterate. A detailed convergence analysis reveals that global convergence properties of line-search and trust-region methods still hold when the methods are accelerated. The analysis is performed in the general context of optimization on manifolds, of which optimization in R n is a particular case. This general convergence analysis sheds a new light on the behavior of several existing algorithms.

show abstract

Globally Convergent Optimization Algorithms on Riemannian Manifolds: Uniform Framework for Unconstrained and Constrained Optimization

Cited by 61 publications

References 15 publications

Geometry of nonlinear least squares with applications to sloppy models and optimization

Geometry of nonlinear least squares with applications to sloppy models and optimization

Trust-Region Methods on Riemannian Manifolds

Accelerated Line-search and Trust-region Methods

Contact Info

Product

Resources

About