Accelerating Deep Learning Training Through Transparent Storage Tiering

Abstract

We present MONARCH, a framework-agnostic storage middleware that transparently employs storage tiering to accelerate Deep Learning (DL) training. It leverages existing storage tiers of modern supercomputers (i.e., compute node's local storage and shared parallel file system (PFS)), while considering the I/O patterns of DL frameworks to improve data placement across tiers. MONARCH aims at accelerating DL training and decreasing the I/O pressure imposed over the PFS. We apply MONARCH to TensorFlow and PyTorch, while validating its performance and applicability under different models and dataset sizes. Results show that, even when the training dataset can only be partially stored at local storage, MONARCH reduces TensorFlow's and PyTorch's training time by up to 28% and 37% for I/O-intensive models, respectively. Furthermore, MONARCH decreases the number of I/O operations submitted to the PFS by up to 56%.

Protecting Metadata Servers From Harm Through Application-level I/O Control

“…I/O optimizations. Many works propose I/O optimizations to reduce the amount of operations submitted to the PFS by resorting to storage tiering [18,30], data reduction techniques [40], and optimized data formats [22,35]. While these can reduce the I/O pressure imposed over the PFS, they still expose it to burstiness and unfairness, since I/O workflows are not rate limited.…”