Convexity of the objective function often allows to guarantee much better convergence rates of iterative minimization methods than in the general non-convex case. However, many problems encountered in training neural networks are non-convex. Some of them satisfy conditions weaker than convexity, but which are still sufficient to guarantee the convergence of some first-order methods.In this work we present a condition to replace convexity and show that gradient descent with fixed step length retains its convergence rate under this condition. We show that the sequential subspace optimization method is optimal in terms of oracle complexity in this case. We also provide a substitute for strong convexity which is sufficient to guarantee the same convergence rate as in the strongly convex case for this new class of generally non-convex functions.

Motivated by recent increased interest in optimization algorithms for nonconvex optimization in application to training deep neural networks and other optimization problems in data analysis, we give an overview of recent theoretical results on global performance guarantees of optimization algorithms for non-convex optimization. We start with classical arguments showing that general non-convex problems could not be solved efficiently in a reasonable time. Then we give a list of problems that can be solved efficiently to find the global minimizer by exploiting the structure of the problem as much as it is possible. Another way to deal with non-convexity is to relax the goal from finding the global minimum to finding a stationary point or a local minimum. For this setting, we first present known results for the convergence rates of deterministic first-order methods, which are then followed by a general theoretical analysis of optimal stochastic and randomized gradient schemes, and an overview of the stochastic first-order methods. After that, we discuss quite general classes of non-convex problems, such as minimization of αweakly-quasi-convex functions and functions that satisfy Polyak-Łojasiewicz condition, which still allow obtaining theoretical convergence guarantees of first-order methods. Then we consider higher-order and zeroth-order/derivative-free methods and their convergence rates for non-convex optimization problems.

No abstract

In this paper, a new variant of accelerated gradient descent is proposed. The proposed method does not require any information about the objective function, uses exact line search for the practical accelerations of convergence, converges according to the well-known lower bounds for both convex and non-convex objective functions, possesses primal-dual properties and can be applied in the non-euclidian set-up. As far as we know this is the first such method possessing all of the above properties at the same time. We also present a universal version of the method which is applicable to non-smooth problems. We demonstrate how in practice one can efficiently use the combination of line-search and primal-duality by considering a convex optimization problem with a simple structure (for example, linearly constrained).