HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism

Abstract

Deep Neural Network (DNN) models have continuously been growing in size in order to improve the accuracy and quality of the models. Moreover, for training of large DNN models, the use of heterogeneous GPUs is inevitable due to the short release cycle of new GPU architectures. In this paper, we investigate how to enable training of large DNN models on a heterogeneous GPU cluster that possibly includes whimpy GPUs that, as a standalone, could not be used for training. We present a DNN training system, HetPipe (Heterogeneous Pipeline), that integrates pipelined model parallelism (PMP) with data parallelism (DP). In HetPipe, a group of multiple GPUs, called a virtual worker, processes minibatches in a pipelined manner, and multiple such virtual workers employ data parallelism for higher performance. We also propose a novel parameter synchronization model, which we refer to as Wave Synchronous Parallel (WSP) to accommodate both PMP and DP for virtual workers, and provide convergence proof of WSP. Our experimental results on a given heterogeneous setting show that with HetPipe, DNN models converge up to 49% faster compared to the state-of-the-art DP technique.

Automatic Graph Partitioning for Very Large-scale Deep Learning

“…Parameter staleness-free Mesh-TensorFlow [14], Megatron-LM [10] Tensor Yes Manual No Yes OptCNN [15], FlexFlow [16], Tofu [17] Tensor Yes Auto No Yes GPipe [11] Graph No Manual No Yes AMPNet [18], XPipe [19] Graph No Manual No No PipeDream [8], SpecTrain [20] Graph Yes Auto No No PipeDream-2BW [21], HetPipe [22] Graph Yes Auto Yes No RaNNC (Ours) Graph Yes Auto Yes Yes In graph partitioning, such tasks are regarded as atomic and cannot be further partitioned. Unfortunately, when the partitioned subcomponents to be computed on different accelerator devices have sequential dependencies, only one accelerator device can be used at a time.…”

“…As mentioned in the previous section, some previous works [18], [19], [8], [20], [21], [22] employed asynchronous pipeline parallelism, which suffers from parameter staleness issues [9]. Such issues are caused by computing a mini-batch using different versions of parameters across stages.…”

“…Since it is often challenging to manually determine stages for pipeline parallelism to achieve a high throughput, some previous works proposed automatic approaches that search for combinations of subcomponents that form stages whose computation times are balanced [20], [8], [21], [22].…”

“…If the given blocks are imbalanced, the entire pipeline suffers from a low throughput due to a timeconsuming bottleneck stage composed of the blocks. Although PipeDream-2BW and HetPipe [22] take layers that repetitively appear as blocks, in their actual models, such intuitively selected layers are not well-balanced. For example, the last layer of the BERT-Based [1] model takes 40% of the overall computation time because it performs matrix multiplication of huge matrices to compute the probabilities of the words in the vocabulary.…”