Quantized Frank-Wolfe: Faster Optimization, Lower Communication, and Projection Free

Abstract

How can we efficiently mitigate the overhead of gradient communications in distributed optimization? This problem is at the heart of training scalable machine learning models and has been mainly studied in the unconstrained setting. In this paper, we propose Quantized Frank-Wolfe (QFW), the first projection-free and communication-efficient algorithm for solving constrained optimization problems at scale. We consider both convex and non-convex objective functions, expressed as a finite-sum or more generally a stochastic optimization problem, and provide strong theoretical guarantees on the convergence rate of QFW. This is accomplished by proposing novel quantization schemes that efficiently compress gradients while controlling the noise variance introduced during this process. Finally, we empirically validate the efficiency of QFW in terms of communication and the quality of returned solution against natural baselines.

Efficient Projection-Free Online Methods with Stochastic Recursive Gradient

“…In the first experiment, we consider an OCO problem -online multiclass logistic regression (Zhang et al 2019) f t as the multiclass logistic loss function…”

“…In the second experiment, we focus on training a onehidden-layer neural network with an additional ℓ 1 norm constraint (Zhang et al 2019). Specifically, given a multiclass data set {(a i , y i )} n i=1 with (a i , y i ) ∈ R d × {1, .…”