Gradient-Free Optimization for Non-Smooth Saddle Point Problems under Adversarial Noise

Dvinskikh, Darina; Tominin, Vladislav; Tominin, Yaroslav; Gasnikov, Alexander

doi:10.48550/arxiv.2202.06114

Cited by 1 publication

(1 citation statement)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A similar approach, utilizing different zeroth-order oracles, was developed and analyzed in [46]. In general, there is a plethora of zeroth-order optimization algorithms, and the interested reader is referred to [4,10,15,26,34], and the references therein.…”

Section: Introductionmentioning

confidence: 99%

A Zeroth-order Proximal Stochastic Gradient Method for Weakly Convex Stochastic Optimization

Pougkakiotis¹,

Kalogerias²

2022

Preprint

View full text Add to dashboard Cite

In this paper we present and analyze a zeroth-order proximal stochastic gradient method suitable for the minimization of weakly convex stochastic optimization problems. We consider non-smooth and nonlinear stochastic composite problems, for which (sub-)gradient information might be unavailable. The proposed algorithm utilizes the well-known Gaussian smoothing technique, which yields unbiased zeroth-order gradient estimators of a related partially smooth surrogate problem. This allows us to employ a standard proximal stochastic gradient scheme for the approximate solution of the surrogate problem, which is determined by a single smoothing parameter, and without the utilization of first-order information. We provide state-of-the-art convergence rates for the proposed zeroth-order method, utilizing a much simpler analysis that requires less restrictive assumptions, as compared with a double Gaussian smoothing alternative recently analyzed in the literature. The proposed method is numerically compared against its (sub-)gradient-based counterparts to demonstrate its viability on a standard phase retrieval problem. Further, we showcase the usefulness and effectiveness of our method for the unique setting of automated hyper-parameter tuning. In particular, we focus on automatically tuning the parameters of optimization algorithms by minimizing a novel heuristic model. The proposed approach is tested on a proximal alternating direction method of multipliers for the solution of L 1 /L 2 -regularized PDE-constrained optimal control problems, with evident empirical success.

show abstract

Section: Introductionmentioning

confidence: 99%