Inexact Model: A Framework for Optimization and Variational Inequalities

Abstract

In this paper we propose a general algorithmic framework for first-order methods in optimization in a broad sense, including minimization problems, saddle-point problems and variational inequalities. This framework allows to obtain many known methods as a special case, the list including accelerated gradient method, composite optimization methods, level-set methods, proximal methods. The idea of the framework is based on constructing an inexact model of the main problem component, i.e. objective function in optimization or operator in variational inequalities. Besides reproducing known results, our framework allows to construct new methods, which we illustrate by constructing a universal method for variational inequalities with composite structure. This method works for smooth and non-smooth problems with optimal complexity without a priori knowledge of the problem smoothness. We also generalize our framework for strongly convex objectives and strongly monotone variational inequalities.

Inexact Tensor Methods and Their Application to Stochastic Convex Optimization

“…Our analysis is based on inexact versions of tensor methods for convex optimization, which use inexact derivatives of higher-order. First-order methods with inexact gradients are well-developed in the literature, see, for example, [62,16,17,21,29,14,20,69] and references therein. Some results on inexact second-order methods are listed above.…”

Gradient Methods for Problems with Inexact Model of the Objective

“…Further, we define Bregman divergence V [y](x) := d(x) − d(y) − ∇d(y), x − y . Next we define the inexact model of the objective function, which generalizes the inexact oracle of [19] (see also [24,10,28,35,60,62]). Definition 1.…”

“…In this subsection we describe a gradient-type method for problems with (δ, L)model of the objective. This algorithm is a natural extension of gradient method, see [35,60,62].…”

Ускоренный И Неускореный Стохастический Градиентный Спуск В Модельной Общности

“…Результаты раздела 2 можно воспроизвести и в модельной общности [1,2,9]. Будем говорить, что функция ψ δ (y, x) является (δ, L)-моделью целевой функции f (x), если для всех x, y ∈ Q функция ψ δ (y, x) -выпукла по y, ψ δ (x, x) ≡ 0,…”

“…С другой стороны известно (см. [1,2,4,5,9]), что если для задачи (1.1) доступен неточный градиент ∇ δ f (x), удовлетворяющий для всех x, y ∈ Q ослабленному условию L-Липшицевости градиента…”