Revisiting Projection-Free Optimization for Strongly Convex Constraint Sets

Rector-Brooks, Jarrid; Wang, Jun-Kun; Mozafari, Barzan

doi:10.1609/aaai.v33i01.33011576

Cited by 4 publications

(3 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The strong convexity of decision sets has been widely utilized in previous studies (Levitin and Polyak, 1966;Dunn, 1979;Garber and Hazan, 2015;Rector-Brooks et al, 2019;Kerdreux et al, 2021). Common examples of strongly convex sets include various balls induced by p norms, Schatten norms, and group norms (Garber and Hazan, 2015).…”

Section: Preliminariesmentioning

confidence: 99%

Online Frank-Wolfe with Unknown Delays

Wan¹,

Tu²,

Zhang³

2022

Preprint

View full text Add to dashboard Cite

The online Frank-Wolfe (OFW) method has gained much popularity for online convex optimization due to its projection-free property. Previous studies showed that for convex losses, OFW attains O(T 3/4 ) regret over general sets and O(T 2/3 ) regret over strongly convex sets, and if losses are strongly convex, these bounds can be improved to O(T 2/3 ) and O( √ T ), respectively. However, they assumed that each gradient queried by OFW is revealed immediately, which may not hold in practice. In this paper, we consider a more practical setting where gradients arrive with arbitrary and unknown delays, and propose delayed OFW which generalizes OFW to this setting. The main idea is to perform an update similar to OFW after receiving any gradient, and play the latest decision for each round. We first show that for convex losses, delayed OFW achieves O(T 3/4 + dT 1/4 ) regret over general sets and O(T 2/3 + dT 1/3 ) regret over strongly convex sets, where d is the maximum delay. Furthermore, we prove that for strongly convex losses, delayed OFW attains O(T 2/3 + d log T ) regret over general sets and O( √ T + d log T ) regret over strongly convex sets. Compared with regret bounds in the non-delayed setting, our results imply that the proposed method is robust to a relatively large amount of delay.

show abstract

Section: Preliminariesmentioning

confidence: 99%

Online Frank-Wolfe with Unknown Delays

Wan¹,

Tu²,

Zhang³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Then, we introduce the definition of the strongly convex set, which has been well studied in offline optimization (Levitin and Polyak 1966;Demyanov and Rubinov 1970;Dunn 1979;Garber and Hazan 2015;Rector-Brooks, Wang, and Mozafari 2019).…”

Section: Preliminariesmentioning

confidence: 99%

Projection-free Online Learning over Strongly Convex Sets

Wan

Zhang

2021

AAAI

View full text Add to dashboard Cite

To efficiently solve online problems with complicated constraints, projection-free algorithms including online frank-wolfe (OFW) and its variants have received significant interest recently. However, in the general case, existing efficient projection-free algorithms only achieved the regret bound of O(T^{3/4}), which is worse than the regret of projection-based algorithms, where T is the number of decision rounds. In this paper, we study the special case of online learning over strongly convex sets, for which we first prove that OFW can enjoy a better regret bound of O(T^{2/3}) for general convex losses. The key idea is to refine the decaying step-size in the original OFW by a simple line search rule. Furthermore, for strongly convex losses, we propose a strongly convex variant of OFW by redefining the surrogate loss function in OFW. We show that it achieves a regret bound of O(T^{2/3}) over general convex sets and a better regret bound of O(T^{1/2}) over strongly convex sets.

show abstract

“…Other projection-free algorithms exist with improved guarantees on strongly convex sets, e.g., for nonconvex optimization [RBWM19], min-max problems [GJLJ17,WA18] or approximate Carathéodory results [CP19]. The various equivalent definitions of strongly convex sets have also stimulated an interest in designing and analysing affine-invariant first-order methods.…”

Section: Introductionmentioning

confidence: 99%

Local and Global Uniform Convexity Conditions

Kerdreux,

d'Aspremont,

Pokutta

2021

Preprint

View full text Add to dashboard Cite

We review various characterizations of uniform convexity and smoothness on norm balls in finitedimensional spaces and connect results stemming from the geometry of Banach spaces with scaling inequalities used in analyzing the convergence of optimization methods. In particular, we establish local versions of these conditions to provide sharper insights on a recent body of complexity results in learning theory, online learning, or offline optimization, which rely on the strong convexity of the feasible set. While they have a significant impact on complexity, these strong convexity or uniform convexity properties of feasible sets are not exploited as thoroughly as their functional counterparts, and this work is an effort to correct this imbalance. We conclude with some practical examples in optimization and machine learning where leveraging these conditions and localized assumptions lead to new complexity results.

show abstract

Revisiting Projection-Free Optimization for Strongly Convex Constraint Sets

Cited by 4 publications

References 3 publications

Online Frank-Wolfe with Unknown Delays

Online Frank-Wolfe with Unknown Delays

Projection-free Online Learning over Strongly Convex Sets

Local and Global Uniform Convexity Conditions

Contact Info

Product

Resources

About