Performance prediction of deep learning applications training in GPU as a service systems

Abstract

growth rate of over 38% to support 3D models, animated video processing, and gaming. GPUaaS adoption will be also boosted by the use of graphics processing units (GPUs) to support Deep learning (DL) model training. Indeed, nowadays, the main cloud providers already offer in their catalogs GPU-based virtual machines pre-installed with the popular DL framework (like Torch, PyTorch, TensorFlow, and Caffe) simplifying DL model programming operations.Motivated by these considerations, this paper studies GPU-deployed neural networks (NNs) and tackles the issue of performance prediction, particularly with respect to NN training times. The proposed approach is based on machine learning and exploits two main sets of features which describe, on one hand, the network architecture and the hyper-parameters, on the other, the hardware characteristics of the target deployment. Such data enable the learning of multiple linear regression models, which, coupled with an established feature selection technique, become accurate prediction tools, with errors below 11 % on average. An extensive experimental campaign, performed both on public and in-house private cloud deployments, considers popular deep NNs used for image classification and speech transcription and shows that prediction errors remain small even when extrapolating outside the range spanned by the input data. This has important implications for the models' applicability: in this way, it is possible to investigate the impact on the performance of different GPUaaS deployment or hardware upgrades even without conducting an empirical investigation on the specific target device or to evaluate the changes in training time when the number of inner modules in the deep neural networks varies.

A Path Relinking Method for the Joint Online Scheduling and Capacity Allocation of DL Training Workloads in GPU as a Service Systems

“…In this work, ML models are adopted to correlate job and target VM features with the expected execution time. In particular, as discussed in [20], linear regression can be used to automatically build models that infer a deep network training time on a particular type of VM. A di erent prediction model is built in [20] for each type of neural network.…”

“…In particular, as discussed in [20], linear regression can be used to automatically build models that infer a deep network training time on a particular type of VM. A di erent prediction model is built in [20] for each type of neural network. The considered features related to the characteristics of the training jobs are the number of iterations and the batch size.…”

“…These numerical features and their inverses are combined to build an extended set that is filtered through sequential forward feature selection [21] and used to feed the ML models. It is worth noting that, since the models consider the number of iterations of the training process, they can be used not only to estimate the execution time for the full execution of the job, but also to predict the time required to complete the remaining iterations of an already started job, even if it is migrated to a di erent VM type [20], [22].…”

“…We considered several instances of the listed DL training workloads, with di erent epochs and batch sizes and, therefore, di erent expected execution times, estimated via the Machine Learning models presented in [22] and discussed in Section 2.2. Such models have been trained using data from profiling runs of the target applications, with an average percentage error below 11% (for detailed accuracy results, see [20]).…”

MALIBOO: When Machine Learning meets Bayesian Optimization

“…Their remarkable predicting capabilities can assist the user in accurately predicting resource usage, execution times, etc. Indeed, previous work [11]- [17] has shown that ML models are usually able to predict these target quantities with very small validation error.…”