Optimizing execution for pipelined‐based distributed deep learning in a heterogeneously networked GPU cluster

Abstract

Exorbitant resources (computing and memory) are required to train a deep neural network (DNN). Often researchers deploy an approach that uses distributed parallel training to acquire larger models faster on GPUs. This approach has its detriments, though; on one hand, a GPU's expanded capacity to compute also produces bigger bottlenecks in inter-GPU's communications during model training, and multi-GPU systems lead to complex connectivity. Workload schedulers then end up having to consider hardware topology and requirements for workload communication, in hopes of allocating GPU resources to optimize execution time and improve usage in a heterogeneous environment. On the other hand, the high memory requirements to train a DNN model make running the training processes on GPUs onerous. To contend with this, we introduce two execution optimization methods based on pipeline-hybrid parallelism (using both data and model parallelism) in a GPU cluster with heterogeneous networking. First, we propose a model partition algorithm that accelerates pipeline-hybrid parallelism training between heterogeneously network-connected GPUs. Second, we introduce a cost-balanced recomputing algorithm to reduce memory usage in the pipeline mode. Experiments show that our solution (Pipe-Torch) averages a speedup of 1.4× compared with data parallelism, and reduces the memory footprint while maintaining pipelined load-balanced training. K E Y W O R D S distributed deep learning, efficient GPU memory, heterogeneous network environment, pipeline-hybrid parallelism 1 INTRODUCTION The last 10 years has witnessed rapid transformation in deep learning, including image recognition, 1 speech recognition, 2 language translation, 3 text identification 4 and clinical medicine. 5,6 Most inroads and successes are attributed to deep neural network (DNN) models' capacity to extract and discern complex features from massive data, while also factoring in that having increased-sized models promotes far greater accuracy. However, the increasing scale of training models-along with the rise in data-remarkably extends the training time. So, while on one hand, these distributed deep-learning frameworks are designed to parallel train DNN models across multiple machines to accelerate training, such as TensorFlow, 7 PyTorch, 8 MxNet; 9 on the other hand, most leading IT (hardware) companies are committed to designing specialized hardware equipment for training (eg, Nvidia GPU 10). However, even with parallel training and special hardware to accelerate training, training a DNN model is time-and memory-intensive. It takes eight Nvidia TITAN X GPUs about 5 days to complete merely one 90-epoch Vgg16 training execution on the ImageNet-1k data set. 11 In terms of

Comparative Study on Distributed Lightweight Deep Learning Models for Road Pothole Detection

“…In addition, two rpc.remote functions are used to call the two shards on two RPC workers so they can be accessed in the forward pass. [31,32].…”