Gradient-only approaches to avoid spurious local minima in unconstrained optimization

Wilke, Daniel N.; Kok, Schalk; Snyman, J. A.; Groenwold, Albert A.

doi:10.1007/s11081-011-9178-7

Cited by 10 publications

(25 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the associated derivative [18] is defined everywhere and it is given by the left hand limit for lower semicontinuous functions and by the right hand limit for upper semi-continuous functions at the discontinuity. The benefit of the complex-step derivative is that it computes the associated derivative and allows the computation of sensitivity information even at a discontinuity.…”

Section: Sensitivities Of Discontinuous Functionsmentioning

confidence: 99%

“…Similarly, derivative information can also always be computed when using (semi)-analytical or automatic differentiation strategies. The computed sensitivities are the associated derivative [18] that can be used for gradient-only optimization.…”

Section: Sensitivities Of Discontinuous Functionsmentioning

confidence: 99%

“…The associated gradient is given by the Calculus gradient when the gradient exists. Where the Calculus gradient does not exist, it is defined by only the left or right sided limit as detailed in [18]. In this study we highlight the connection between the complex-step derivative and the associated derivative.…”

Section: Introductionmentioning

confidence: 98%

“…Since non-differentiable functions are not everywhere differentiable, the concept of subgradients was introduced to allow the gradient field to be defined everywhere. Recently, Wilke et al [18] proposed gradient-only optimization as an alternative approach to mathematical programming to solve discontinuous optimization problems, which are non-differentiable and not continuous. The gradient field is defined everywhere via associated gradients, which allows sensitivity information to be defined at a discontinuity.…”

Section: Introductionmentioning

confidence: 99%

“…Positive projection points are characterized by a positive directional derivative at all points in a small ball around such a point. The direction vector is defined by the point in the ball where the directional derivative is to be computed and the positive projection point itself [18].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Numerical Sensitivity Computation for Discontinuous Gradient-Only Optimization Problems Using the Complex-Step Method

Wilke¹,

Kok²

2014

Proceedings of 10th World Congress on Computational Mechanics

View full text Add to dashboard Cite

Abstract. This study considers the numerical sensitivity calculation for discontinuous gradientonly optimization problems using the complex-step method. The complex-step method was initially introduced to differentiate analytical functions in the late 1960s, and is based on a Taylor series expansion using a pure imaginary step. The complex-step method is not subject to subtraction errors as with finite difference approaches when computing first order sensitivities and therefore allows for much smaller step sizes that ultimately yields accurate sensitivities. This study investigates the applicability of the complex-step method to numerically compute first order sensitivity information for discontinuous optimization problems. An attractive feature of the complex-step approach is that no real difference step is taken as with conventional finite difference approaches, since conventional finite differences are problematic when real steps are taken over a discontinuity. We highlight the benefits and disadvantages of the complex-step method in the context of discontinuous gradient-only optimization problems that result from numerically approximated (partial) differential equations.Gradient-only optimization is a recently proposed alternative to mathematical programming for solving discontinuous optimization problems. Gradient-only optimization was initially proposed to solve shape optimization problems that utilise remeshing (i.e. the mesh topology is allowed to change) between design updates. Here, changes in mesh topology result in abrupt changes in the discretization error of the computed response. These abrupt changes in turn manifests as discontinuities in the numerically computed objective and constraint functions of an optimization problem. These discontinuities are in particular problematic when they manifest as local minima. Note that these potential issues are not limited to problems in shape optimization but may be present whenever (partial) differential equations are approximated numerically with non-constant discretization methods e.g. remeshing of spatial domains or automatic time stepping in temporal domains.

show abstract

Section: Sensitivities Of Discontinuous Functionsmentioning

confidence: 99%

Section: Sensitivities Of Discontinuous Functionsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 98%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Numerical Sensitivity Computation for Discontinuous Gradient-Only Optimization Problems Using the Complex-Step Method

Wilke¹,

Kok²

2014

Proceedings of 10th World Congress on Computational Mechanics

View full text Add to dashboard Cite

show abstract

Relaxed error control in shape optimization that utilizes remeshing

Wilke

Kok

Groenwold

2013

Numerical Meth Engineering

Self Cite

View full text Add to dashboard Cite

SUMMARYShape optimization strategies based on error indicators usually require strict error control for every computed design during the optimization run. The strict error control serves two purposes. Firstly, it allows for the accurate computation of the structural response used to define the shape optimization problem itself. Secondly, it reduces the discretization error, which in turn reduces the size of the step discontinuities in the objective function that result from remeshing in the first place. These discontinuities may trap conventional optimization algorithms, which rely on both function and gradient evaluations, in local minima. This has the drawback that multiple analyses and error computations are often required per design to control the error.In this study we propose a methodology that relaxes the requirements for strict error control for each design. Instead, we rather control the error as the iterations progress. Our approach only requires a single analysis and error computation per design. Consequently, large discontinuities may initially be accommodated; their intensities reduce as the iterations progress. To circumvent the difficulties associated with local minima due to remeshing, we rely on gradient-only optimization algorithms, which have previously been shown to be able to robustly overcome these discontinuities.

show abstract

Resolving learning rates adaptively by locating stochastic non-negative associated gradient projection points using line searches

Kafka

Wilke

2020

J Glob Optim

View full text Add to dashboard Cite

Gradient-only approaches to avoid spurious local minima in unconstrained optimization

Cited by 10 publications

References 18 publications

Numerical Sensitivity Computation for Discontinuous Gradient-Only Optimization Problems Using the Complex-Step Method

Numerical Sensitivity Computation for Discontinuous Gradient-Only Optimization Problems Using the Complex-Step Method

Relaxed error control in shape optimization that utilizes remeshing

Resolving learning rates adaptively by locating stochastic non-negative associated gradient projection points using line searches

Contact Info

Product

Resources

About