STRONGHOLD: Fast and Affordable Billion-Scale Deep Learning Model Training

Abstract

Deep neural networks (DNNs) with billion-scale parameters have demonstrated impressive performance in solving many tasks. Unfortunately, training a billion-scale DNN is out of the reach of many data scientists because it requires highperformance GPU servers that are too expensive to purchase and maintain. We present STRONGHOLD, a novel approach for enabling large DNN model training with no change to the user code. STRONGHOLD scales up the largest trainable model size by dynamically offloading data to the CPU RAM and enabling the use of secondary storage. It automatically determines the minimum amount of data to be kept in the GPU memory to minimize GPU memory usage. Compared to state-of-the-art offloading-based solutions, STRONGHOLD improves the trainable model size by 1.9x∼6.5x on a 32GB V100 GPU, with 1.2x∼3.7x improvement on the training throughput. It has been deployed into production to successfully support large-scale DNN training.

Training and Serving System of Foundation Models: A Comprehensive Survey

“…Stronghold [58] introduces a work window method, which keeps only part of the model's layers and parameters in the GPU. Under this mechanism, the GPU processes only the model layers within the work window, transferring the rest to the CPU.…”