Crossbow

Abstract

Deep learning models are trained on servers with many GPUs, and training must scale with the number of GPUs. Systems such as TensorFlow and Caffe2 train models with parallel synchronous stochastic gradient descent: they process a batch of training data at a time, partitioned across GPUs, and average the resulting partial gradients to obtain an updated global model. To fully utilise all GPUs, systems must increase the batch size, which hinders statistical efficiency. Users tune hyper-parameters such as the learning rate to compensate for this, which is complex and model-specific. We describe CROSSBOW, a new single-server multi-GPU system for training deep learning models that enables users to freely choose their preferred batch size-however small-while scaling to multiple GPUs. CROSSBOW uses many parallel model replicas and avoids reduced statistical efficiency through a new synchronous training method. We introduce SMA, a synchronous variant of model averaging in which replicas independently explore the solution space with gradient descent, but adjust their search synchronously based on the trajectory of a globally-consistent average model. CROSSBOW achieves high hardware efficiency with small batch sizes by potentially training multiple model replicas per GPU, automatically tuning the number of replicas to maximise throughput. Our experiments show that CROSSBOW improves the training time of deep learning models on an 8-GPU server by 1.3-4× compared to TensorFlow.

Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers

“…Once this is done, the next epoch can begin. The synchronization imposed by gradient aggregation at every epoch is the main limitation of synchronous SGD-known as the straggler problem [27]. Asynchronous SGD [37,42,39,32,2] transforms gradient aggregation into a completely asynchronous process in which a GPU transitions to the next epoch immediately after its partial gradient is added to the aggregated gradient.…”

“…However, beyond a certain point, a large learning rate impacts model convergence negatively [19]. Thus, model averaging is a more reliable algorithm [27].…”

“…Elastic model averaging is implemented only in CNTK. In CROSSBOW [27], every GPU has a local replica of the model, while the global model -which is the average of the local replicas -can be either mirrored across GPUs or centrally stored on CPU. Similar to elastic model averaging, the local replicas are allowed to evolve independently.…”

“…As these two values increase, building accurate deep learning models becomes time-consuming even on specialized hardware accelerators such as GPUs [19] and TPUs [48] because of their reduced memory-which requires many slow data transfers with the CPU [9]. Thus, in order to scale training, parallel processing across multiple GPUs [12,27] becomes a necessity. This approach is facilitated by the preponderance of multi-GPU computing architectures both on supercomputers [58] and in the cloud [52].…”

“…In a typical dataset (Table 1), the feature vector, as well as the associated labels, are highly sparse, thus processing requires sparse linear algebra operations. This is quite different from the image classification benchmarks -which are based on dense linear algebra -used almost exclusively to evaluate the existing multi-GPU training algorithms [19,49,27]. AttentionXML [47], DeepXML [13], and LightXML [25] are some of the most recent deep learning XML algorithms that achieve the highest accuracy.…”