The Case for Storage Optimization Decoupling in Deep Learning Frameworks

Macedo, Ricardo; Correia, Cláudia Rafaela Gomes; Dantas, Marco; Brito, Claúdia; Xu, Wei; Tanimura, Yusuke; Haga, Jason; Paulo, João

doi:10.1109/cluster48925.2021.00096

Cited by 6 publications

(4 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Results of each experiment concern the average and standard deviation of 7 runs. For the RestNet-50, all setups perform similarly, ranging from 64 and 67 minutes of execution time, as it imposes less I/O demand [15]. Interestingly, the Lustre setup exhibits the highest training time variability across identical runs of each experiment.…”

Section: A DL Training Under Different Storage Setupsmentioning

confidence: 96%

“…CoorDL [17] provides insights on storage I/O data stalls and mitigates them by providing a new in-memory caching policy. PRISMA [15] proposes a Software-Defined Storage data plane that performs data prefetching to memory.…”

Section: Related Workmentioning

confidence: 99%

“…Storage tiering systems, such as Hermes [20], are focused towards buffering scientific write workloads at intermediary storage mediums before reaching the PFS. However, DL training workloads are read-oriented, and have specific I/O patterns that should be considered when optimizing data placement over different storage tiers [15], [21]. Namely, the full dataset must be accessed for each training epoch, and each dataset file is read once per epoch.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Accelerating Deep Learning Training Through Transparent Storage Tiering

Dantas

Leitao

Cui

et al. 2022

2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid)

Self Cite

View full text Add to dashboard Cite

We present MONARCH, a framework-agnostic storage middleware that transparently employs storage tiering to accelerate Deep Learning (DL) training. It leverages existing storage tiers of modern supercomputers (i.e., compute node's local storage and shared parallel file system (PFS)), while considering the I/O patterns of DL frameworks to improve data placement across tiers. MONARCH aims at accelerating DL training and decreasing the I/O pressure imposed over the PFS. We apply MONARCH to TensorFlow and PyTorch, while validating its performance and applicability under different models and dataset sizes. Results show that, even when the training dataset can only be partially stored at local storage, MONARCH reduces TensorFlow's and PyTorch's training time by up to 28% and 37% for I/O-intensive models, respectively. Furthermore, MONARCH decreases the number of I/O operations submitted to the PFS by up to 56%.

show abstract

Section: A DL Training Under Different Storage Setupsmentioning

confidence: 96%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Accelerating Deep Learning Training Through Transparent Storage Tiering

Dantas

Leitao

Cui

et al. 2022

2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid)

Self Cite

View full text Add to dashboard Cite

show abstract

“…For the purpose of this work, we assume the preprocessing is performed on the CPU, while the training is performed on the GPUs. This is a common scenario [2,19,21,25,26,42] that makes efficient use of heterogeneous compute resources.…”

Section: Introductionmentioning

confidence: 99%

Lobster: Load Balance-Aware I/O for Distributed DNN Training

Liu

Nicolae

Liu

2022

Proceedings of the 51st International Conference on Parallel Processing

View full text Add to dashboard Cite

The resource-hungry and time-consuming process of training Deep Neural Networks (DNNs) can be accelerated by optimizing and/or scaling computations on accelerators such as GPUs. However, the loading and pre-processing of training samples then often emerges as a new bottleneck. This data loading process engages a complex pipeline that extends from the sampling of training data on external storage to delivery of those data to GPUs, and that comprises not only expensive I/O operations but also decoding, shuffling, batching, augmentation, and other operations. We propose in this paper a new holistic approach to data loading that addresses three challenges not sufficiently addressed by other methods: I/O load imbalances among the GPUs on a node; rigid resource allocations to data loading and data preprocessing steps, which lead to idle resources and bottlenecks; and limited efficiency of caching strategies based on pre-fetching due to eviction of training samples needed soon at the expense of those needed later. We first present a study of key bottlenecks observed as training samples flow through the data loading and preprocessing pipeline. Then, we describe Lobster, a data loading runtime that uses performance modeling and advanced heuristics to combine flexible thread management with optimized eviction for distributed caching in order to mitigate I/O overheads and load imbalances. Experiments with a range of models and datasets show that the Lobster approach reduces both I/O overheads and end-to-end training times by up to 1.5× compared with stateof-the-art approaches.

show abstract