Evaluation of Deep Learning Frameworks Over Different HPC Architectures

Shams, Soheil; Platania, Richard; Lee, Kisung; Park, Seung-Jong

doi:10.1109/icdcs.2017.259

Cited by 59 publications

(25 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, employing NVLink (either P2P or CL) introduces additional overhead (e.g., enable/disable peer access, routing, NCCL initialization, etc). To summarize, a faster GPU interconnect such as NVLink has been reported to be beneficial for accelerating modern deep-learning frameworks [31], [32]. However, regarding general GPGPU applications, without (i) replacing the underlying CPU-centric master-slave programming model by a more distributed parallelization model, or (ii) migrating the communication master role to a GPU (e.g., offloading GPU communication control from CPU to GPU via techniques such as NVSHMEM [33]), optimized inter-GPU communication via faster intra-node GPU interconnect such as NVLinks can hardly become significant enough to lift the entire application's speedup.…”

Section: Impact Of Nvlinkmentioning

confidence: 99%

Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect

Song

Chen

et al. 2020

IEEE Trans. Parallel Distrib. Syst.

164

View full text Add to dashboard Cite

High performance multi-GPU computing becomes an inevitable trend due to the ever-increasing demand on computation capability in emerging domains such as deep learning, big data and planet-scale simulations. However, the lack of deep understanding on how modern GPUs can be connected and the real impact of state-of-the-art interconnect technology on multi-GPU application performance become a hurdle. In this paper, we fill the gap by conducting a thorough evaluation on five latest types of modern GPU interconnects: PCIe, NVLink-V1, NVLink-V2, NVLink-SLI and NVSwitch, from six high-end servers and HPC platforms: NVIDIA P100-DGX-1, V100-DGX-1, DGX-2, OLCF's SummitDev and Summit supercomputers, as well as an SLI-linked system with two NVIDIA Turing RTX-2080 GPUs. Based on the empirical evaluation, we have observed four new types of GPU communication network NUMA effects: three are triggered by NVLink's topology, connectivity and routing, while one is caused by PCIe chipset design issue. These observations indicate that, for an application running in a multi-GPU node, choosing the right GPU combination can impose considerable impact on GPU communication efficiency, as well as the application's overall performance. Our evaluation can be leveraged in building practical multi-GPU performance models, which are vital for GPU task allocation, scheduling and migration in a shared environment (e.g., AI cloud and HPC centers), as well as communication-oriented performance tuning.

show abstract

Section: Impact Of Nvlinkmentioning

confidence: 99%

Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect

Song

Chen

et al. 2020

IEEE Trans. Parallel Distrib. Syst.

164

View full text Add to dashboard Cite

show abstract

“…Shams et al in [14] already performed a study of HPC clusters with machine learning framework. However, they include accelerators (KNL and GPU), and they do not include considerations about the effects of optimized linear algebra routines.…”

Section: Related and Future Workmentioning

confidence: 99%

TensorFlow on State-of-the-Art HPC Clusters: A Machine Learning use Case

Ramirez-Gargallo

García-Gasulla

Mantovani

2019

2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)

View full text Add to dashboard Cite

The recent rapid growth of the data-flow programming paradigm enabled the development of specific architectures, e.g., for machine learning. The most known example is the Tensor Processing Unit (TPU) by Google. Standard data-centers, however, still can not foresee large partitions dedicated to machine learning specific architectures. Within data-centers, the High-Performance Computing (HPC) clusters are highly parallel machines targeting a broad class of compute-intensive workflows, as such they can be used for tackling machine learning challenges. On top of this, HPC architectures are rapidly changing, including accelerators and instruction sets other than the classical x86 CPUs. In this blurry scenario, identifying which are the best hardware/software configurations to efficiently support machine learning workloads on HPC clusters is not trivial. In this paper, we considered the workflow of TensorFlow for image recognition. We highlight the strong dependency of the performance in the training phase on the availability of arithmetic libraries optimized for the underlying architecture. Following the example of Intel leveraging the MKL libraries for improving the TensorFlow performance, we plugged the Arm Performance Libraries into TensorFlow and tested on an HPC cluster based on Marvell ThunderX2 CPUs. Also, we performed a scalability study on three state-of-the-art HPC clusters based on different CPU architectures, x86 Intel Skylake, Arm-v8 Marvell ThunderX2, and PowerPC IBM Power9.

show abstract

“…In addition, TensorFlow [24] and Caffe [25] can also be used for distributed training. The performance of distributed training frameworks has been studied on the high-performance computing (HPC) architecture, in a previous study [26]. This study analyzed the training time of Caffe, TensorFlow, and SINGA with respect to the CPU and GPU configurations in HPC, using an Intel Xeon and IBM Power 8 Processor.…”

Section: Related Workmentioning

confidence: 99%

Hardware Resource Analysis in Distributed Training with Edge Devices

2019

View full text Add to dashboard Cite

When training a deep learning model with distributed training, the hardware resource utilization of each device depends on the model structure and the number of devices used for training. Distributed training has recently been applied to edge computing. Since edge devices have hardware resource limitations such as memory, there is a need for training methods that use hardware resources efficiently. Previous research focused on reducing training time by optimizing the synchronization process between edge devices or by compressing the models. In this paper, we monitored hardware resource usage based on the number of layers and the batch size of the model during distributed training with edge devices. We analyzed memory usage and training time variability as the batch size and number of layers increased. Experimental results demonstrated that, the larger the batch size, the fewer synchronizations between devices, resulting in less accurate training. In the shallow model, training time increased as the number of devices used for training increased because the synchronization between devices took more time than the computation time of training. This paper finds that efficient use of hardware resources for distributed training requires selecting devices in the context of model complexity and that fewer layers and smaller batches are required for efficient hardware use.

show abstract

Evaluation of Deep Learning Frameworks Over Different HPC Architectures

Cited by 59 publications

References 8 publications

Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect

Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect

TensorFlow on State-of-the-Art HPC Clusters: A Machine Learning use Case

Hardware Resource Analysis in Distributed Training with Edge Devices

Contact Info

Product

Resources

About