Revisiting Small Batch Training for Deep Neural Networks

Abstract

Modern deep neural network training is typically based on mini-batch stochastic gradient optimization. While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance and allows a significantly smaller memory footprint, which might also be exploited to improve machine throughput. In this paper, we review common assumptions on learning rate scaling and training duration, as a basis for an experimental comparison of test performance for different mini-batch sizes. We adopt a learning rate that corresponds to a constant average weight update per gradient calculation (i.e., per unit cost of computation), and point out that this results in a variance of the weight updates that increases linearly with the mini-batch size m. The collected experimental results for the CIFAR-10, CIFAR-100 and ImageNet datasets show that increasing the mini-batch size progressively reduces the range of learning rates that provide stable convergence and acceptable test performance. On the other hand, small mini-batch sizes provide more up-to-date gradient calculations, which yields more stable and reliable training. The best performance has been consistently obtained for mini-batch sizes between m = 2 and m = 32, which contrasts with recent work advocating the use of mini-batch sizes in the thousands.

On Large-Cohort Training for Federated Learning

“…Moreover, largecohort training can introduce fundamental optimization and generalization issues. Our results are reminiscent of work on large-batch training in centralized settings, where larger batches can stagnate convergence improvements (Dean et al, 2012;You et al, 2017;Golmant et al, 2018;McCandlish et al, 2018;Yin et al, 2018), and even lead to generalization issues with deep neural networks (Shallue et al, 2019;Ma et al, 2018;Keskar et al, 2017;Hoffer et al, 2017;Masters and Luschi, 2018;Lin et al, 2019Lin et al, , 2020. While some of the challenges we identify with large-cohort training are parallel to issues that arise in large-batch centralized learning, others are unique to federated learning and have not been previously identified in the literature.…”

“…This property of diminishing returns has been explored both empirically (Dean et al, 2012;McCandlish et al, 2018;Golmant et al, 2018;Shallue et al, 2019) and theoretically (Ma et al, 2018;Yin et al, 2018). Beyond the issue of speedup saturation, numerous works have also observed a generalization gap when training deep neural networks with large batches (Keskar et al, 2017;Hoffer et al, 2017;You et al, 2017;Masters and Luschi, 2018;Lin et al, 2019Lin et al, , 2020. Our work differs from these areas by specifically exploring how the cohort size (the number of selected clients) affects federated optimization methods.…”

“…Large-batch centralized optimization methods have repeatedly been shown to converge to models with worse generalization ability than models found by small-batch methods (Keskar et al, 2017;Hoffer et al, 2017;You et al, 2017;Masters and Luschi, 2018;Lin et al, 2019Lin et al, , 2020. Given the parallels between batch size in centralized learning and cohort size in FL, this raises obvious questions about whether similar issues occur in FL.…”

Learning phase field mean curvature flows with neural networks

“…We now turn to the question of how the learning parameters of our networks are estimated. The learning procedure consists in seeking an optimal vector parameter θ minimizing the energy J 1 (or J k , k > 1 depending on the problem) using a mini-batch stochastic gradient descent algorithm with adaptive momentum [55,56,57] on a set of training data.…”

Consistency Regularization for Deep Face Anti-Spoofing

“…Different from [9], we don't use synchronized batch normalization (BN) but standard BN in our experiment, and we find that synchronized BN brings a little damage to the performance. The possible reason is that large batch size when synchronizing BN will result in local optimal solution [44], especially for the FAS task in our experiments.…”