The Convergence of Stochastic Gradient Descent in Asynchronous Shared Memory

Abstract

Stochastic Gradient Descent (SGD) is a fundamental algorithm in machine learning, representing the optimization backbone for training several classic models, from regression to neural networks. Given the recent practical focus on distributed machine learning, significant work has been dedicated to the convergence properties of this algorithm under the inconsistent and noisy updates arising from execution in a distributed environment. However, surprisingly, the convergence properties of this classic algorithm in the standard shared-memory model are still not well-understood.In this work, we address this gap, and provide new convergence bounds for lock-free concurrent stochastic gradient descent, executing in the classic asynchronous shared memory model, against a strong adaptive adversary. Our results give improved upper and lower bounds on the "price of asynchrony" when executing the fundamental SGD algorithm in a concurrent setting. They show that this classic optimization tool can converge faster and with a wider range of parameters than previously known under asynchronous iterations. At the same time, we exhibit a fundamental trade-off between the maximum delay in the system and the rate at which SGD can converge, which governs the set of parameters under which this algorithm can still work efficiently.

Game of Threads

“…For example, if both threads sample the same θ and additionally sample minibatches that are 100% biased towards the same label, the gradients derived by both threads will be maximally similar. 6 We show evidence to support this claim in the evaluation. Specifically, we show that as minibatch bias increases, less attack threads are required to move the model out of its converged state.…”

“…In the A-SGD literature, the closest work to that presented in our paper is [6]. That work discusses how an adversary can slow convergence by influencing scheduling.…”

“…For example, the focused attack in Section 4.2 relies on a specific minibatch, but applies that minibatch to up-to-date model parameters. Thus, that attack implicitly pauses the attack thread in between Lines 9 and 11 6. Note that while gradient directions will be similar, they will not be identical because even with the same θ , the exact minibatch contents will be different.…”

MindTheStep-AsyncPSGD: Adaptive Asynchronous Parallel Stochastic Gradient Descent

“…The results in Theorem 6 and Corollary 3 are related to the results presented in [10] and [4]. The main differences are that in our analysis we tighten the bound with a factor (2 − θ) −1 , expand the allowed step size interval, as well as relax the maximum staleness assumption and reduce the magnitude of the bound from linear in the maximum staleness O(τ ) to the expected O(τ ).…”

“…Properties of Async-PSGD with sparse or componentwise updates have since been rigourously studied in recent literature due to the performance benefits of lockfreedom [28][24] [10]. The gradient sparsity assumption was relaxed in the recent work [4] which magnified the convergence time bound in the order of magnitude ∼ √ d, d being the problem dimensionality. Delayed optimization in completely asynchronous first-order optimization algorithms was analyzed initially in [2], where Agarwal et al introduce step sizes which diminish over the progression of SGD, depending on the maximum staleness allowed in the system, but not adaptive to the actual delays observed.…”