Software-hardware co-design for fast and scalable training of deep learning recommendation models

Mudigere, Dheevatsa; Hao, Yajiang; Huang, Jianyu; Jia, Zhihao; Tulloch, Andrew; Sridharan, Srinivas; Liu, Xing; Özdal, Mustafa; Nie, Jade; Park, Jongsoo; Luo, Liang; Yang, Jie; Gao, Leon; Ivchenko, Dmytro; Basant, Aarti; Hu, Yuxi; Yang, Jiyan; Ardestani, Ehsan K.; Wang, Xiaodong; Komuravelli, Rakesh; Chu, C. Y. Cyrus; Yılmaz, Serhat; Li, Huayu; Qian, Jiyuan; Feng, Zhuobo; Ma, Yinbin; Yang, Junjie; Wen, Ellie; Li, Hong; Yang, Lin; Sun, Cuicui; Zhao, Whitney; Melts, Dimitry; Dhulipala, Krishna; Kishore, K. R.; Graf, Tyler; Eisenman, Assaf; Matam, Kiran Kumar; Gangidi, Adi; Chen, Guoqiang Jerry; Krishnan, Manojkumar; Nayak, Avinash; Nair, Krishnakumar; Muthiah, Bharath; khorashadi, Mahmoud; Bhattacharya, P.; Lapukhov, Petr; Naumov, Maxim; Mathews, Ajit; Qiao, Lin; Smelyanskiy, Mikhail; Jia, Bill; Rao, V. Chandra Shekhar

doi:10.1145/3470496.3533727

“…Applicability of single CPU-GPU ScratchPipe. Recent literature from Facebook elaborate on the scale of their deployed RecSys model size, exhibiting upto 100× difference in number of embedding tables per model [61], [62]. Specifically, RecSys models for content filtering are known to be smaller sized than those for ranking [25], enabling the overall memory footprint to fit within a single-node's CPU memory capacity.…”

Section: G Discussionmentioning

confidence: 99%

Training Personalized Recommendation Systems from (GPU) Scratch: Look Forward not Backwards

Kwon¹,

Rhu²

2022

Preprint

View full text Add to dashboard Cite

Personalized recommendation models (RecSys) are one of the most popular machine learning workload serviced by hyperscalers. A critical challenge of training RecSys is its high memory capacity requirements, reaching hundreds of GBs to TBs of model size. In RecSys, the so-called embedding layers account for the majority of memory usage so current systems employ a hybrid CPU-GPU design to have the large CPU memory store the memory hungry embedding layers. Unfortunately, training embeddings involve several memory bandwidth intensive operations which is at odds with the slow CPU memory, causing performance overheads. Prior work proposed to cache frequently accessed embeddings inside GPU memory as means to filter down the embedding layer traffic to CPU memory, but this paper observes several limitations with such cache design. In this work, we present a fundamentally different approach in designing embedding caches for RecSys. Our proposed ScratchPipe architecture utilizes unique properties of RecSys training to develop an embedding cache that not only sees the past but also the "future" cache accesses. ScratchPipe exploits such property to guarantee that the active working set of embedding layers can "always" be captured inside our proposed cache design, enabling embedding layer training to be conducted at GPU memory speed.

show abstract

“…Applicability of single CPU-GPU ScratchPipe. Recent literature from Facebook elaborate on the scale of their deployed RecSys model size, exhibiting upto 100× difference in number of embedding tables per model [14,39]. Specifically, RecSys models for content filtering are known to be smaller sized than those for ranking [15], enabling the overall memory footprint to fit within a singlenode's CPU memory capacity.…”

Section: Discussionmentioning

confidence: 99%

Training personalized recommendation systems from (GPU) scratch

Kwon

¹

,

Rhu

²

2022

Proceedings of the 49th Annual International Symposium on Computer Architecture

View full text Add to dashboard Cite

Personalized recommendation models (RecSys) are one of the most popular machine learning workload serviced by hyperscalers. A critical challenge of training RecSys is its high memory capacity requirements, reaching hundreds of GBs to TBs of model size. In RecSys, the so-called embedding layers account for the majority of memory usage so current systems employ a hybrid CPU-GPU design to have the large CPU memory store the memory hungry embedding layers. Unfortunately, training embeddings involve several memory bandwidth intensive operations which is at odds with the slow CPU memory, causing performance overheads. Prior work proposed to cache frequently accessed embeddings inside GPU memory as means to filter down the embedding layer traffic to CPU memory, but this paper observes several limitations with such cache design. In this work, we present a fundamentally different approach in designing embedding caches for RecSys. Our proposed ScratchPipe architecture utilizes unique properties of RecSys training to develop an embedding cache that not only sees the past but also the "future" cache accesses. ScratchPipe exploits such property to guarantee that the active working set of embedding layers can "always" be captured inside our proposed cache design, enabling embedding layer training to be conducted at GPU memory speed.

show abstract

“…Our production recommendation models are built on the opensource DLRM architecture [63]. Modern DLRMs are massive, consisting of over 12 trillion parameters to train, requiring ≈ 1 zetaFLOPs of total compute [59]. To meet the compute requirements of DLRM training, we built the ZionEX hardware platform.…”

Section: Recommendation Model Backgroundmentioning

confidence: 99%

“…A node also contains four CPU sockets, each with a dedicated 100 Gbps NIC connected to our regular datacenter network for data ingestion. We defer readers to [59] for more details on ZionEX.…”

Section: Recommendation Model Backgroundmentioning

confidence: 99%

“…For example, the latest MLPerf Training round (v1.1) [56] contains submissions from Azure and NVIDIA using 2048 and 4320 A100 GPUs, respectively, whereas Google submitted training results using pods containing up to 4096 TPUv4s [58]. At Meta, we are building datacenter-scale AI training clusters by both scaling our production datacenters to include thousands of GPUs using ZionEX nodes [59] and building the AI Research Super-Cluster (RSC) with over ten thousand GPUs [10]. These DSAs have been laser-focused on optimizing and scaling compute for training, namely matrix-heavy computations used during backpropagation.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Understanding data storage and ingestion for large-scale deep recommendation model training

Zhao¹,

Agarwal²,

Basant³

et al. 2022

Proceedings of the 49th Annual International Symposium on Computer Architecture

Self Cite

View full text Add to dashboard Cite

Datacenter-scale AI training clusters consisting of thousands of domain-specific accelerators (DSA) are used to train increasinglycomplex deep learning models. These clusters rely on a data storage and ingestion (DSI) pipeline, responsible for storing exabytes of training data and serving it at tens of terabytes per second. As DSAs continue to push training efficiency and throughput, the DSI pipeline is becoming the dominating factor that constrains the overall training performance and capacity. Innovations that improve the efficiency and performance of DSI systems and hardware are urgent, demanding a deep understanding of DSI characteristics and infrastructure at scale. This paper presents Meta's end-to-end DSI pipeline, composed of a central data warehouse built on distributed storage and a Data PreProcessing Service that scales to eliminate data stalls. We characterize how hundreds of models are collaboratively trained across geo-distributed datacenters via diverse and continuous training jobs. These training jobs read and heavily filter massive and evolving datasets, resulting in popular features and samples used across training jobs. We measure the intense network, memory, and compute resources required by each training job to preprocess samples during training. Finally, we synthesize key takeaways based on our production infrastructure characterization. These include identifying hardware bottlenecks, discussing opportunities for heterogeneous DSI hardware, motivating research in datacenter scheduling and benchmark datasets, and assimilating lessons learned in optimizing DSI infrastructure. CCS CONCEPTS• Software and its engineering → Distributed systems organizing principles; • Information systems → Database management system engines; • Computing methodologies → Machine learning.

show abstract

Software-hardware co-design for fast and scalable training of deep learning recommendation models

Cited by 61 publications

References 48 publications

Training Personalized Recommendation Systems from (GPU) Scratch: Look Forward not Backwards

Training Personalized Recommendation Systems from (GPU) Scratch: Look Forward not Backwards

Training personalized recommendation systems from (GPU) scratch

Understanding data storage and ingestion for large-scale deep recommendation model training

Contact Info

Product

Resources

About