Layer-Centric Memory Reuse and Data Migration for Extreme-Scale Deep Learning on Many-Core Architectures

Abstract

Due to the popularity of Deep Neural Network (DNN) models, we have witnessed extreme-scale DNN models with the continued increase of the scale in terms of depth and width. However, the extremely high memory requirements for them make it difficult to run the training processes on single many-core architectures such as a Graphic Processing Unit (GPU), which compels researchers to use model parallelism over multiple GPUs to make it work. However, model parallelism always brings very heavy additional overhead. Therefore, running an extreme-scale model in a single GPU is urgently required. There still exist several challenges to reduce the memory footprint for extreme-scale deep learning. To address this tough problem, we first identify the memory usage characteristics for deep and wide convolutional networks, and demonstrate the opportunities for memory reuse at both the intra-layer and inter-layer levels. We then present Layrub, a runtime data placement strategy that orchestrates the execution of the training process. It achieves layer-centric reuse to reduce memory consumption for extreme-scale deep learning that could not previously be run on a single GPU. Experiments show that, compared to the original Caffe, Layrub can cut down the memory usage rate by an average of 58.2% and by up to 98.9%, at the moderate cost of 24.1% higher training execution time on average. Results also show that Layrub outperforms some popular deep learning systems such as GeePS, vDNN, MXNet, and Tensorflow. More importantly, Layrub can tackle extreme-scale deep learning tasks. For example, it makes an extra-deep ResNet with 1,517 layers that can be trained successfully in one GPU with 12GB memory, while other existing deep learning systems cannot.

Layup

“…We do the evaluation in the following three aspects: First, the execution time potential of Layup is evaluated, comparing with some existing state-of-the-art works, including vDNN [32], Layrub [25], Caffe [24], and SuperNeurons [42]. Since SuperNeurons is the most effective approach among them and was put forward recently, we carry out a separate comparison with this technique in more detail.…”

“…AlexNet is used for this comparison. Both of these memory optimizations are evaluated based on their best performance implementations [4,25] on Caffe. As shown in Figures 2(a) and 2(b), we make the following findings: the CPU-GPU transfer outperforms the extra forward computation in the CONV1-CONV5 and FC6-FC8 layers (by an average speedup of 7.4×), but underperforms for the rest of the layers.…”

“…These approaches use prefetching and offloading techniques to hide the communication cost. Layrub [25] shows that feature maps and gradient maps in the training process can be reused from algorithmic and architectural perspectives. The primary limitation of this work is that its inter-layer memory Layup: Memory Optimization for GPU-based CNNs 39:5 reuse strongly depends on the transfer bandwidth between CPU and GPU, which may introduce significant costs and long delays due to bandwidth contention.…”

“…Gradient map: There are already several existing methods that try to optimize the memory usage for this type of intermediate data, and the intra-layer strategy of Layrub [25] is typical of these. The gradient map reuses the memory space of the corresponding feature map.…”

“…Reusing feature maps (a type of intermediate data) to save memory usage is a classic method that has been studied in many existing works. vDNN [32], GeePS [11], and Layrub [25] utilize CPU memory as a limitless assistant memory for the temporary storage of intermediate data. By using a CPU-GPU transfer, the memory usage in the GPU can be significantly decreased.…”