Hippie: A Data-Paralleled Pipeline Approach to Improve Memory-Efficiency and Scalability for Large DNN Training

Ye, Xiangyu; Lai, Zhiquan; Li, Shengwei; Cai, Lei; Sun, Ding; Qiao, Linbo; Li, Dongsheng

doi:10.1145/3472456.3472497

Cited by 13 publications

(2 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To tackle the issue mentioned above, a model named Hippie was proposed [26]. Hippie is a hybrid parallel training framework that combines pipeline and data parallelism to increase the memory economy and scalability of massive DNN training.…”

Section: Related Workmentioning

confidence: 99%

Comparative Study on Distributed Lightweight Deep Learning Models for Road Pothole Detection

Tahir

Jung

2023

Sensors

View full text Add to dashboard Cite

This paper delves into image detection based on distributed deep-learning techniques for intelligent traffic systems or self-driving cars. The accuracy and precision of neural networks deployed on edge devices (e.g., CCTV (closed-circuit television) for road surveillance) with small datasets may be compromised, leading to the misjudgment of targets. To address this challenge, TensorFlow and PyTorch were used to initialize various distributed model parallel and data parallel techniques. Despite the success of these techniques, communication constraints were observed along with certain speed issues. As a result, a hybrid pipeline was proposed, combining both dataset and model distribution through an all-reduced algorithm and NVlinks to prevent miscommunication among gradients. The proposed approach was tested on both an edge cluster and Google cluster environment, demonstrating superior performance compared to other test settings, with the quality of the bounding box detection system meeting expectations with increased reliability. Performance metrics, including total training time, images/second, cross-entropy loss, and total loss against the number of the epoch, were evaluated, revealing a robust competition between TensorFlow and PyTorch. The PyTorch environment’s hybrid pipeline outperformed other test settings.

show abstract

Section: Related Workmentioning

confidence: 99%

Comparative Study on Distributed Lightweight Deep Learning Models for Road Pothole Detection

Tahir

Jung

2023

Sensors

View full text Add to dashboard Cite

show abstract

“…DAPPLE [12] and Megatron-LM [29] modify the pipeline schedule and reduce the usage of activation memory. Synchronous PMP works suffer from bubbles, Hippie [48] utilizes bubbles for communications with half step delayed parameter updating, and Chimera [22] uses a bidirectional pipeline to reduce the bubble but doubles the model memory.…”

Section: Background and Related Workmentioning

confidence: 99%

Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models

Lai¹,

Li²,

Tang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Foundation models are becoming the dominant deep learning technologies. Pretraining a foundation model is always time-consumed due to the large scale of both the model parameter and training dataset. Besides being computing-intensive, the training process is extremely memory-intensive and communication-intensive. These features make it necessary to apply 3D parallelism, which integrates data parallelism, pipeline model parallelism and tensor model parallelism, to achieve high training efficiency.To achieve this goal, some custom software frameworks such as Megatron-LM and DeepSpeed are developed. However, current 3D parallelism frameworks still meet two issues: i) they are not transparent to model developers, which need to manually modify the model to parallelize training. ii) their utilization of computation, GPU memory and network bandwidth are not sufficient. We propose Merak, an automated 3D parallelism deep learning training framework with high resource utilization. Merak automatically deploys with an automatic model partitioner, which uses a graph sharding algorithm on a proxy representation of the model. Merak also presents the non-intrusive API for scaling out foundation model training with minimal code modification. In addition, we design a high-performance 3D parallel runtime engine in Merak. It uses several techniques to exploit available training resources, including shifted critical path pipeline schedule that brings a higher computation utilization, stage-aware recomputation that makes use of idle worker memory, and sub-pipelined tensor model parallelism that overlaps communication and computation. Experiments on 64 GPUs show Merak can speedup the training performance over the state-of-the-art 3D parallelism frameworks of models with 1.5, 2.5, 8.3, and 20 billion parameters by up to 1.42×, 1.39×, 1.43×, and 1.61×, respectively.

show abstract

CSIMD: Cross-Search Algorithm with Improved Multi-dimensional Dichotomy for Micro-Batch-Based Pipeline Parallel Training in DNN

Zhou,

Lan,

Xie

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Hippie: A Data-Paralleled Pipeline Approach to Improve Memory-Efficiency and Scalability for Large DNN Training

Cited by 13 publications

References 19 publications

Comparative Study on Distributed Lightweight Deep Learning Models for Road Pothole Detection

Comparative Study on Distributed Lightweight Deep Learning Models for Road Pothole Detection

Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models

CSIMD: Cross-Search Algorithm with Improved Multi-dimensional Dichotomy for Micro-Batch-Based Pipeline Parallel Training in DNN

Contact Info

Product

Resources

About