A Variable Batch Size Strategy for Large Scale Distributed DNN Training

Hu, Zhongzhe; Xiao, Junmin; Tian, Zhongbo; Zhang, Xiaoyang; Zhu, Hongrui; Yao, Chengji; Sun, Ninghui; Tan, Guangming

doi:10.1109/ispa-bdcloud-sustaincom-socialcom48970.2019.00074

Cited by 10 publications

(7 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…As training is scaled up to 128 GPUs across 64 nodes, all models shift to lower memory utilization; however, we see that the GNNs also see lower SM utilization alongside memory utilization. The decrease in memory utilization observed across models with an increasing number of GPUs is in line with our expectation that batch size scaling is still important for efficient distributed training [16]. Something to note is that while BERT sees about a two-fold reduction in memory utilization (from about 40% to 20%) as we scale up training from 2 to 128 GPUs, SM utilization barely changes (and slightly increases).…”

Section: Experimental Results and Findingssupporting

confidence: 82%

Benchmarking Resource Usage for Efficient Distributed Deep Learning

Frey¹,

Li²,

McDonald³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep learning (DL) workflows demand an everincreasing budget of compute and energy in order to achieve outsized gains. Neural architecture searches, hyperparameter sweeps, and rapid prototyping consume immense resources that can prevent resource-constrained researchers from experimenting with large models and carry considerable environmental impact. As such, it becomes essential to understand how different deep neural networks (DNNs) and training leverage increasing compute and energy resources-especially specialized computationally-intensive models across different domains and applications.In this paper, we conduct over 3,400 experiments training an array of deep networks representing various domains/tasks-natural language processing, computer vision, and chemistry-on up to 424 graphics processing units (GPUs). During training, our experiments systematically vary compute resource characteristics and energy-saving mechanisms such as power utilization and GPU clock rate limits to capture and illustrate the different trade-offs and scaling behaviors each representative model exhibits under various resource and energyconstrained regimes. We fit power law models that describe how training time scales with available compute resources and energy constraints. We anticipate that these findings will help inform and guide highperformance computing providers in optimizing resource utilization, by selectively reducing energy consumption for different deep learning tasks/workflows with minimal impact on training.

show abstract

Section: Experimental Results and Findingssupporting

confidence: 82%

Benchmarking Resource Usage for Efficient Distributed Deep Learning

Frey¹,

Li²,

McDonald³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Three convolutional layers and a fully connected layer are selected. The x‐axis represents the number of training epochs 10 …”

Section: Variable Batch Size Strategymentioning

confidence: 99%

“…Top‐1 training error and test (validation) error of ResNet‐50 network trained using ImageNet‐1K dataset with different batch size 10 …”

Section: Introductionmentioning

confidence: 99%

Fast and accurate variable batch size convolution neural network training on large scale distributed systems

Xiao

Sun

et al. 2022

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

Large-scale distributed convolution neural network (CNN) training brings two performance challenges: model performance and system performance. Large batch size usually leads to model test accuracy loss, which counteracts the benefits of parallel SGD. The existing solutions require massive hyperparameter hand-tuning. To overcome this difficult, we analyze the training process and find that earlier training stages are more sensitive to batch size. Accordingly, we assert that different stages should use different batch size, and propose a variable batch size strategy. In order to remain high test accuracy under larger batch size cases, we design an auto-tuning engine for automatic parameter tuning in the proposed variable batch size strategy. Furthermore, we develop a dataflow implementation approach to achieve the high-throughput CNN training on supercomputer system. Our approach has achieved high generalization performance on SOAT CNN networks. For the ShuffleNet, training with ImageNet-1K dataset, we scale the batch size to 120 K without accuracy loss and to 128 K with only a slight loss. And the dataflow implementation approach achieves 93.5% scaling efficiency on 1024 GPUs compared with the state-of-the-art.

show abstract

“…Вступ. За останнє десятиліття, набір алгоритмів машинного навчання під назвою глибоке навчання (deep learning) призвів до значних поліпшень у задачах комп'ютерного зору [10], розпізнавання та обробки природних мов [2,3]. Це призвело до широкого застосування різноманітних комерційних продуктів, що засновані на навчанні, в різних сферах людської діяльності.…”

unclassified

“…У кількох останніх роботах обговорюється використання великої швидкості навчання [7], малого [8] та великого [9,10] розміру підвиборок наборів навчальних даних. Вони демонструють, що співвідношення швидкості навчання до розміру підвиборки набору даних впливає на навчання.…”

unclassified

An Approach to Accelerate the Training of Convolutional Neural Networks by Tuning the Hyperparameters of Learning

Радюк¹

2020

CSIT

View full text Add to dashboard Cite

Over the last decade, a set of machine learning algorithms called deep learning has led to significant improvements in computer vision, natural language recognition and processing. This has led to the widespread use of a variety of commercial, learning-based products in various fields of human activity. Despite this success, the use of deep neural networks remains a black box. Today, the process of setting hyperparameters and designing a network architecture requires experience and a lot of trial and error and is based more on chance than on a scientific approach. At the same time, the task of simplifying deep learning is extremely urgent. To date, no simple ways have been invented to establish the optimal values of learning hyperparameters, namely learning speed, sample size, data set, learning pulse, and weight loss. Grid search and random search of hyperparameter space are extremely resource intensive. The choice of hyperparameters is critical for the training time and the final result. In addition, experts often choose one of the standard architectures (for example, ResNets and ready-made sets of hyperparameters. However, such kits are usually suboptimal for specific practical tasks. The presented work offers an approach to finding the optimal set of hyperparameters of learning ZNM. An integrated approach to all hyperparameters is valuable because there is an interdependence between them. The aim of the work is to develop an approach for setting a set of hyperparameters, which will reduce the time spent during the design of ZNM and ensure the efficiency of its work. In recent decades, the introduction of deep learning methods, in particular convolutional neural networks (CNNs), has led to impressive success in image and video processing. However, the training of CNN has been commonly mostly based on the employment of quasi-optimal hyperparameters. Such an approach usually requires huge computational and time costs to train the network and does not guarantee a satisfactory result. However, hyperparameters play a crucial role in the effectiveness of CNN, as diverse hyperparameters lead to models with significantly different characteristics. Poorly selected hyperparameters generally lead to low model performance. The issue of choosing optimal hyperparameters for CNN has not been resolved yet. The presented work proposes several practical approaches to setting hyperparameters, which allows reducing training time and increasing the accuracy of the model. The article considers the function of training validation loss during underfitting and overfitting. There are guidelines in the end to reach the optimization point. The paper also considers the regulation of learning rate and momentum to accelerate network training. All experiments are based on the widespread CIFAR-10 and CIFAR-100 datasets.

show abstract

A Variable Batch Size Strategy for Large Scale Distributed DNN Training

Cited by 10 publications

References 13 publications

Benchmarking Resource Usage for Efficient Distributed Deep Learning

Benchmarking Resource Usage for Efficient Distributed Deep Learning

Fast and accurate variable batch size convolution neural network training on large scale distributed systems

An Approach to Accelerate the Training of Convolutional Neural Networks by Tuning the Hyperparameters of Learning

Contact Info

Product

Resources

About