Accelerated Gradient Methods Combining Tikhonov Regularization with Geometric Damping Driven by the Hessian

Attouch, Hédy; Balhag, Aicha; Chbani, Zaki; Riahi, Hassan

doi:10.1007/s00245-023-09997-x

Cited by 12 publications

(8 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We are going to show global convergence rates in expectation in the smooth convex case, in order to do that, it is worth citing a result from [10], where they deal with the deterministic case.…”

Section: Convergence Rates Of the Objective In The Smooth Casementioning

confidence: 99%

Tikhonov Regularization for Stochastic Non-Smooth Convex Optimization in Hilbert Spaces

Maulen-Soto,

Fadili,

Attouch

2024

Preprint

Self Cite

View full text Add to dashboard Cite

To solve non-smooth convex optimization problems with a noisy gradient input, we analyze the global behavior of subgradient-like flows under stochastic errors. The objective function is composite, being equal to the sum of two convex functions, one being differentiable and the other potentially non-smooth. We then use stochastic differential inclusions where the drift term is minus the subgradient of the objective function, and the diffusion term is either bounded or square-integrable. In this context, under Lipschitz's continuity of the differentiable term and a growth condition of the non-smooth term, our first main result shows almost sure weak convergence of the trajectory process towards a minimizer of the objective function. Then, using Tikhonov regularization with a properly tuned vanishing parameter, we can obtain almost sure strong convergence of the trajectory towards the minimum norm solution. We find an explicit tuning of this parameter when our objective function satisfies a local error-bound inequality. We also provide a comprehensive complexity analysis by establishing several new pointwise and ergodic convergence rates in expectation for the convex and strongly convex case. AMS subject classifications. 37N40, 46N10, 49M99, 65B99, 65K05, 65K10, 90B50, 90C25, 60H10, 49J52.

show abstract

Section: Convergence Rates Of the Objective In The Smooth Casementioning

confidence: 99%

Tikhonov Regularization for Stochastic Non-Smooth Convex Optimization in Hilbert Spaces

Maulen-Soto,

Fadili,

Attouch

2024

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The main goal of the research is to provide the setting in a nonsmooth case, where we would have fast convergence of the function values combined with strong convergence of the trajectories to the element of the minimal norm from the set of all minimizers of the objective function. This analysis is an extrapolation of the one conducted in [ 3 ] to the case of a nonsmooth objective function. We also aim to provide the exact rates of convergence of the values for the polynomial choice of the smoothing parameter and Tikhonov function .…”

Section: Introductionmentioning

confidence: 96%

“…The objective function is no longer required to be (continuously) differentiable, which gives us more freedom in choosing the latter. Moreover, we show that the main quantities , , and go to zero, as , without specifying (as it was done in [ 3 ]) the choice of the functions and . We are also able to obtain rates of convergence of function values in case of the polynomial choice of parameters for , which is not an option in [ 3 ].…”

Section: Introductionmentioning

confidence: 96%

“…One of the fine examples in a smooth setting is presented below (see [ 3 ]) where , is twice continuously differentiable and convex, is nonincreasing and goes to zero, as , and p is chosen appropriately. This system inherits the properties of fast convergence rates of the function values, being of the order , and additionally provides the strong convergence results for the trajectories of the system in the same setting.…”

Section: Introductionmentioning

confidence: 99%

“…In this paper we aim to develop the ideas presented in [ 3 ] for to cover the nonsmooth case. The objective function is no longer required to be (continuously) differentiable, which gives us more freedom in choosing the latter.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A fast continuous time approach for non-smooth convex optimization using Tikhonov regularization technique

Karapetyants

2023

Comput Optim Appl

View full text Add to dashboard Cite

In this paper we would like to address the classical optimization problem of minimizing a proper, convex and lower semicontinuous function via the second order in time dynamics, combining viscous and Hessian-driven damping with a Tikhonov regularization term. In our analysis we heavily exploit the Moreau envelope of the objective function and its properties as well as Tikhonov regularization properties, which we extend to a nonsmooth case. We introduce the setting, which at the same time guarantees the fast convergence of the function (and Moreau envelope) values and strong convergence of the trajectories of the system to a minimal norm solution—the element of the minimal norm of all the minimizers of the objective. Moreover, we deduce the precise rates of convergence of the values for the particular choice of parameters. Various numerical examples are also included as an illustration of the theoretical results.

show abstract

Convex optimization via inertial algorithms with vanishing Tikhonov regularization: fast convergence to the minimum norm solution

Attouch,

László

2024

Math Meth Oper Res

View full text Add to dashboard Cite

In a Hilbertian framework, for the minimization of a general convex differentiable function f, we introduce new inertial dynamics and algorithms that generate trajectories and iterates that converge fastly towards the minimizer of f with minimum norm. Our study is based on the non-autonomous version of the Polyak heavy ball method, which, at time t, is associated with the strongly convex function obtained by adding to f a Tikhonov regularization term with vanishing coefficient $$\varepsilon (t)$$ ε ( t ) . In this dynamic, the damping coefficient is proportional to the square root of the Tikhonov regularization parameter $$\varepsilon (t)$$ ε ( t ) . By adjusting the speed of convergence of $$\varepsilon (t)$$ ε ( t ) towards zero, we will obtain both rapid convergence towards the infimal value of f, and the strong convergence of the trajectories towards the element of minimum norm of the set of minimizers of f. In particular, we obtain an improved version of the dynamic of Su-Boyd-Candès for the accelerated gradient method of Nesterov. This study naturally leads to corresponding first-order algorithms obtained by temporal discretization. In the case of a proper lower semicontinuous and convex function f, we study the proximal algorithms in detail, and show that they benefit from similar properties.

show abstract

Accelerated Gradient Methods Combining Tikhonov Regularization with Geometric Damping Driven by the Hessian

Cited by 12 publications

References 35 publications

Tikhonov Regularization for Stochastic Non-Smooth Convex Optimization in Hilbert Spaces

Tikhonov Regularization for Stochastic Non-Smooth Convex Optimization in Hilbert Spaces

A fast continuous time approach for non-smooth convex optimization using Tikhonov regularization technique

Convex optimization via inertial algorithms with vanishing Tikhonov regularization: fast convergence to the minimum norm solution

Contact Info

Product

Resources

About