A parallel sparse tensor benchmark suite on CPUs and GPUs

Li, Jiajia; Lakshminarasimhan, Mahesh; Wu, Xiaolong; Li, Ang; Olschanowsky, Catherine; Barker, Kevin

doi:10.1145/3332466.3374513

Cited by 4 publications

(1 citation statement)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We presume that renting a low-cost VM and profiling the different strategies could probe the infrastructure, i.e., network bandwidths. This allows us to extrapolate the processing performance by tensor-specific CPU benchmarks like PASTA [52] for high-cost VMs. We provide our library as an open-source project at https://github.com/cirquit/presto.…”

Section: Presto Librarymentioning

confidence: 99%

Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines

Isenko,

Mayer,

Jedele

et al. 2022

Preprint

View full text Add to dashboard Cite

Preprocessing pipelines in deep learning aim to provide sufficient data throughput to keep the training processes busy. Maximizing resource utilization is becoming more challenging as the throughput of training processes increases with hardware innovations (e.g., faster GPUs, TPUs, and inter-connects) and advanced parallelization techniques that yield better scalability. At the same time, the amount of training data needed in order to train increasingly complex models is growing. As a consequence of this development, data preprocessing and provisioning are becoming a severe bottleneck in end-to-end deep learning pipelines.In this paper, we provide an in-depth analysis of data preprocessing pipelines from four different machine learning domains. We introduce a new perspective on efficiently preparing datasets for endto-end deep learning pipelines and extract individual trade-offs to optimize throughput, preprocessing time, and storage consumption. Additionally, we provide an open-source profiling library that can automatically decide on a suitable preprocessing strategy to maximize throughput. By applying our generated insights to real-world usecases, we obtain an increased throughput of 3× to 13× compared to an untuned system while keeping the pipeline functionally identical. These findings show the enormous potential of data pipeline tuning.

show abstract

Section: Presto Librarymentioning

confidence: 99%