Characterizing Deep Learning Training Workloads on Alibaba-PAI

Wang, Mengdi; Meng, Chen; Long, Guangcheng; Wu, Chuan; Yang, Jun; Lin, Wei; Jia, Yangqing

doi:10.1109/iiswc47752.2019.9042047

Cited by 34 publications

(21 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…9 Appendix 9.1 PipeDream-2BW Convergence PipeDream-2BW boosts pipeline training throughput by sacrificing sync SGD semantics, which can lead to sub-optimal accuracy results. Although the authors show convergence results for Bert, asynchronous training is not a common practice today, primarily since it is unclear how final accuracy will be affected due to stale backward updates [39].…”

Section: Discussionmentioning

confidence: 98%

Varuna: Scalable, Low-cost Training of Massive Deep Learning Models

Athlur¹,

Saran²,

Sivathanu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Systems for training massive deep learning models (billions of parameters) today assume and require specialized "hyperclusters": hundreds or thousands of GPUs wired with specialized high-bandwidth interconnects such as NV-Link and Infiniband. Besides being expensive, such dependence on hyper-clusters and custom high-speed inter-connects limits the size of such clusters, creating (a) scalability limits on job parallelism; (b) resource fragmentation across hyperclusters.In this paper, we present Varuna a new system that enables training massive deep learning models on commodity networking. Varuna makes thrifty use of networking resources and automatically configures the user's training job to efficiently use any given set of resources. Therefore, Varuna is able to leverage "low-priority" VMs that cost about 5x cheaper than dedicated GPUs, thus significantly reducing the cost of training massive models. We demonstrate the efficacy of Varuna by training massive models, including a 200 billion parameter model, on 5x cheaper "spot VMs", while maintaining high training throughput. Varuna improves end-to-end training time by up to 18x compared to other model-parallel approaches and up to 26% compared to other pipeline parallel approachesThe code for Varuna is available at https://github.com/ microsoft/varuna.

show abstract

Section: Discussionmentioning

confidence: 98%

Varuna: Scalable, Low-cost Training of Massive Deep Learning Models

Athlur¹,

Saran²,

Sivathanu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Xu et al [15] leverage virtualized GPU metrics and vCPU in isolation to propose an approach to predict slowdown from co-located DL workloads. Wang et al [19] obtain DL workload and infrastructure features to determine suitable training regime. Antman [10] also leverages GPU Utilization to first identify jobs that maybe suitable for co-location.…”

Section: Related Workmentioning

confidence: 99%

“…For consumers, this allows greater insight into potential GPU costs 1 . Understanding and exploiting DL workload utilization to improve co-location is critical for designing resourceefficient DL systems [19], [20], [10]. However, established approaches for characterizing GPU utilization from DL workloads leverage online profiling during execution.…”

Section: Introductionmentioning

confidence: 99%

Horus: Interference-Aware and Prediction-Based Scheduling in Deep Learning Systems

Yeung

Borowiec

Yang

et al. 2022

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

To accelerate the training of Deep Learning (DL) models, clusters of machines equipped with hardware accelerators such as GPUs are leveraged to reduce execution time. State-of-the-art resource managers are needed to increase GPU utilization and maximize throughput. While co-locating DL jobs on the same GPU has been shown to be effective, this can incur interference causing slowdown. In this paper we propose Horus: an interference-aware and prediction-based resource manager for DL systems. Horus proactively predicts GPU utilization of heterogeneous DL jobs extrapolated from the DL model's computation graph features, removing the need for online profiling and isolated reserved GPUs. Through micro-benchmarks and job co-location combinations across heterogeneous GPU hardware, we identify GPU utilization as a general proxy metric to determine good placement decisions, in contrast to current approaches which reserve isolated GPUs to perform online profiling and directly measure GPU utilization for each unique submitted job. Our approach promotes high resource utilization and makespan reduction; via real-world experimentation and large-scale trace driven simulation, we demonstrate that Horus outperforms other DL resource managers by up to 61.5% for GPU resource utilization, 23.7-30.7% for makespan reduction and 68.3% in job wait time reduction.

show abstract

“…On the one hand, due to cost, power, and space constraints, edge boxes typically possess weaker GPUs than their cloud counterparts [4, 27,83]. On the other hand, analytics deployments face rapidly increasing workloads due to the following trends: (1) more camera feeds to analyze [27,47,49], (2) more models to run due to increased popularity and shifts to bring-your-own-model platforms [16,23,38,48], and (3) increased model complexity, primarily through growing numbers of layers and parameters (Figure 1) [15,50,51,92]. Taken together, the result is an ever-worsening resource picture for edge video analytics.…”

Section: Introductionmentioning

confidence: 99%

GEMEL: Model Merging for Memory-Efficient, Real-Time Video Analytics at the Edge

Padmanabhan¹,

Agarwal²,

Iyer³

et al. 2022

Preprint

View full text Add to dashboard Cite

Video analytics pipelines have steadily shifted to edge deployments to reduce bandwidth overheads and privacy violations, but in doing so, face an ever-growing resource tension. Most notably, edge-box GPUs lack the memory needed to concurrently house the growing number of (increasingly complex) models for real-time inference. Unfortunately, existing solutions that rely on time/space sharing of GPU resources are insufficient as the required swapping delays result in unacceptable frame drops and accuracy violations. We present model merging, a new memory management technique that exploits architectural similarities between edge vision models by judiciously sharing their layers (including weights) to reduce workload memory costs and swapping delays. Our system, GEMEL, efficiently integrates merging into existing pipelines by ( 1) leveraging several guiding observations about per-model memory usage and interlayer dependencies to quickly identify fruitful and accuracypreserving merging configurations, and (2) altering edge inference schedules to maximize merging benefits. Experiments across diverse workloads reveal that GEMEL reduces memory usage by up to 60.7%, and improves overall accuracy by 8-39% relative to time/space sharing alone.

show abstract

Characterizing Deep Learning Training Workloads on Alibaba-PAI

Cited by 34 publications

References 37 publications

Varuna: Scalable, Low-cost Training of Massive Deep Learning Models

Varuna: Scalable, Low-cost Training of Massive Deep Learning Models

Horus: Interference-Aware and Prediction-Based Scheduling in Deep Learning Systems

GEMEL: Model Merging for Memory-Efficient, Real-Time Video Analytics at the Edge

Contact Info

Product

Resources

About