DIESEL: A Dataset-Based Distributed Storage and Caching System for Large-Scale Deep Learning Training

Wang, Lipeng; Ye, Songgao; Yang, Baichen; Lu, Youyou; Zhang, Hequan; Yan, Shengen; Luo, Qiong

doi:10.1145/3404397.3404472

Cited by 24 publications

(9 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DIESEL [35] and DIESEL+ [36] cache the metadata entirely in memory and cache the loaded data in memory. They double the training performance.…”

Section: Related Workmentioning

confidence: 99%

WITHDRAWN: Efficient Data Loading for Deep Neural Network Training

Liu

Shi

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep neural networks (DNN) have achieved outstanding results in a wide range of applications. However, deep neural network training is a data-intensive and compute-intensive task to obtain high accuracy. While multiple state-of-the-art AI accelerators such as GPUs can be deployed in a single machine for training, data loading for deep neural network training dominates a significant amount of time in the whole training process. As a result, data loading becomes the key bottleneck to limit the whole training performance. This issue is mainly caused by the factor that a large number of small-size files are loaded during the training process. In this paper, we present efficient data loading for DNN training by using two methods: Based on data access pattern and efficient utilization of disk I/O, we use a block as the unit for data reading. As data are repeated used in training an accurate model, we cache partially loaded data for faster access. We use image-net as the dataset and present comprehensive evaluations in terms of running time and accuracy for stateof-the-art deep neural networks on a machine with four V100 GPU cards. The evaluation results demonstrated while that the accuracy is maintained for commonly used DNNs, our optimizations can achieve up to 10 fold performance growth in terms of the whole running time for DNN training.

show abstract

“…DIESEL [35] and DIESEL+ [36] cache the metadata entirely in memory and cache the loaded data in memory. They double the training performance.…”

Section: Related Workmentioning

confidence: 99%

WITHDRAWN: Efficient Data Loading for Deep Neural Network Training

Liu

Shi

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…These characteristics result in limited data reuse opportunities. This is in stark contrast to common benchmark models and datasets which systematically ingest the entire dataset across multiple epochs, and systems that optimize for these characteristics [24,29,36,47].…”

Section: Industry-scale Recsys Trainingmentioning

confidence: 97%

“…Similarly, DeepIO [52] and DLFS [53] leverage hardware support in the form of RDMA and NVMeOF to provide randomized minibatches from storage. DIESEL [47] co-designs storage and caching to provide efficient randomized minibatches for small files. Wang et al [48] and Kumar et al [30] explore mitigating data stalls on TPUs.…”

Section: Related Workmentioning

confidence: 99%

Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training

Zhao,

Agarwal,

Basant

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Other recent works target key DSI components, focusing on benchmark vision and NLP models. CoorDL [57], Quiver [47], and DIESEL [77] are caches that optimize for single-server training, HP tuning jobs, and small files, respectively. DeepIO [87] and DLFS [88] leverage hardware (RDMA and NVMeOF) to provide randomized minibatches from storage.…”

Section: Related Workmentioning

confidence: 99%

Understanding data storage and ingestion for large-scale deep recommendation model training

Zhao¹,

Agarwal²,

Basant³

et al. 2022

Proceedings of the 49th Annual International Symposium on Computer Architecture

View full text Add to dashboard Cite

Datacenter-scale AI training clusters consisting of thousands of domain-specific accelerators (DSA) are used to train increasinglycomplex deep learning models. These clusters rely on a data storage and ingestion (DSI) pipeline, responsible for storing exabytes of training data and serving it at tens of terabytes per second. As DSAs continue to push training efficiency and throughput, the DSI pipeline is becoming the dominating factor that constrains the overall training performance and capacity. Innovations that improve the efficiency and performance of DSI systems and hardware are urgent, demanding a deep understanding of DSI characteristics and infrastructure at scale. This paper presents Meta's end-to-end DSI pipeline, composed of a central data warehouse built on distributed storage and a Data PreProcessing Service that scales to eliminate data stalls. We characterize how hundreds of models are collaboratively trained across geo-distributed datacenters via diverse and continuous training jobs. These training jobs read and heavily filter massive and evolving datasets, resulting in popular features and samples used across training jobs. We measure the intense network, memory, and compute resources required by each training job to preprocess samples during training. Finally, we synthesize key takeaways based on our production infrastructure characterization. These include identifying hardware bottlenecks, discussing opportunities for heterogeneous DSI hardware, motivating research in datacenter scheduling and benchmark datasets, and assimilating lessons learned in optimizing DSI infrastructure. CCS CONCEPTS• Software and its engineering → Distributed systems organizing principles; • Information systems → Database management system engines; • Computing methodologies → Machine learning.

show abstract

DIESEL: A Dataset-Based Distributed Storage and Caching System for Large-Scale Deep Learning Training

Cited by 24 publications

References 12 publications

WITHDRAWN: Efficient Data Loading for Deep Neural Network Training

WITHDRAWN: Efficient Data Loading for Deep Neural Network Training

Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training

Understanding data storage and ingestion for large-scale deep recommendation model training

Contact Info

Product

Resources

About