Measuring the Effects of Data Parallelism on Neural Network Training

Abstract

Recent hardware developments have dramatically increased the scale of data parallelism available for neural network training. Among the simplest ways to harness next-generation hardware is to increase the batch size in standard mini-batch neural network training algorithms. In this work, we aim to experimentally characterize the effects of increasing the batch size on training time, as measured by the number of steps necessary to reach a goal out-of-sample error. We study how this relationship varies with the training algorithm, model, and data set, and find extremely large variation between workloads. Along the way, we show that disagreements in the literature on how batch size affects model quality can largely be explained by differences in metaparameter tuning and compute budgets at different batch sizes. We find no evidence that larger batch sizes degrade out-of-sample performance. Finally, we discuss the implications of our results on efforts to train neural networks much faster in the future. Our experimental data is publicly available as a database of 71,638,836 loss measurements taken over the course of training for 168,160 individual models across 35 workloads.

A Loss Curvature Perspective on Training Instability in Deep Learning

“…A clear picture emerges from these observations. Previous research suggests that in order to effectively leverage larger batch sizes, one has to increase the learning rate in tandem with the batch size [12,7,27,19]. Our results suggest that large values of λ 1 place a sharp limit on the maximum the learning rate possible and therefore, limit the model's ability to leverage data parallelism effectively.…”

“…Previous research has studied the interplay of the loss curvature and batch size scaling from various different perspectives. Most notably, Shallue et al [27] observe that increasing the batch size yields consistent improvements in training speed until a (problem-dependent) critical batch size is reached; increasing the batch size beyond this threshold yields diminishing improvements in training speed. Zhang et al [35] observe that a simple Noisy Quadratic Model (NQM) is able to capture the empirical behavior observed in [27].…”

“…Most notably, Shallue et al [27] observe that increasing the batch size yields consistent improvements in training speed until a (problem-dependent) critical batch size is reached; increasing the batch size beyond this threshold yields diminishing improvements in training speed. Zhang et al [35] observe that a simple Noisy Quadratic Model (NQM) is able to capture the empirical behavior observed in [27]. Similarly, McCandlish et al [19] use quadratic approximations to the loss to provide a closed form expression 94.65 Table 1: A comparison between the GradInit method and Kaiming initialization with gradient clipping on the DenseNet-100 architecture.…”

“…4 We then measure the number of training steps required to reach 85% validation accuracy, and the optimal learning rate found for each batch size. Similar to [27], we normalize the plotted steps to 85% accuracy by the value measured at batch size 64.…”

Batch size-invariance for policy optimization

“…Here we consider behavior as a function of the total number of examples processed, so another way to put this is that doubling the batch size halves the number of steps needed. Shallue et al [2018] and refer to this as "perfect scaling".…”