An Adaptive High Order Method for Finding Third-Order Critical Points of Nonconvex Optimization

Abstract

It is well known that finding a global optimum is extremely challenging for nonconvex optimization. There are some recent efforts [1,[12][13][14] regarding the optimization methods for computing higherorder critical points, which can exclude the so-called degenerate saddle points and reach a solution with better quality. Desipte theoretical development in [1,[12][13][14], the corresponding numerical experiments are missing. In this paper, we propose an implementable higher-order method, named adaptive high order method (AHOM), that aims to find the third-order critical points. This is achieved by solving an "easier" subproblem and incorporating the adaptive strategy of parametertuning in each iteration of the algorithm. The iteration complexity of the proposed method is established. Some preliminary numerical results are provided to show AHOM is able to escape the degenerate saddle points, where the second-order method could possibly get stuck.

Recent Theoretical Advances in Non-Convex Optimization

“…One of the motivations for such methods that the second-order method could get stuck at the so-called degenerate saddle point, where the Hessian matrix has nonnegative eigenvalues with some eigenvalues equal to 0. In paper [263] it is shown how gradient descent and cubic regularization method stuck in such points for even small problems, like f (x, y) = x 3 − 3xy 2 in degenerate saddle point point (0, 0). So, we should use third-order information to escape such points.…”

On complexity and convergence of high-order coordinate descent algorithms for smooth nonconvex box-constrained minimization

“…In this paper we are concerned with complexity issues of CD methods that employ highorder models to approximate the subproblems that arise at each iteration. The use of highorder models for unconstrained optimization was defined and analyzed from the point of view of worst-case complexity in [6] and other subsequent papers [5,21,36,37,45,57]. In [5] numerical implementations with quartic regularization were introduced.…”