<i>iGniter:</i> Interference-Aware GPU Resource Provisioning for Predictable DNN Inference in the Cloud

Xu, Fei; Xu, Jianian; Chen, Jiabin; Chen, Li; Shang, Ruitao; Zhou, Zhi; Liu, Fangming

doi:10.1109/tpds.2022.3232715

Cited by 24 publications

(2 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, Scrooge tackles inference DL workloads. Igniter [22], focused also on inference jobs, is an interference-aware resource provisioning framework that allows spatial GPU sharing and jointly optimizes resource allocation and scheduling. The authors model the performance interference analytically, and Igniter determines the optimal batch size, a lower bound for GPU resources and a greedy placement minimizing the interference.…”

Section: Related Workmentioning

confidence: 99%

“…Note that, while the interference generated by node partitioning can be neglected (see, e.g., [11]- [15]), this is not generally true for GPU sharing. Nevertheless, we decided to consider this scenario, which is currently gaining popularity in the literature [19], [22], since the large memory of the most recent GPUs usually allows to simultaneously execute multiple mini-jobs, freeing resources for more demanding jobs. Moreover, GPU sharing is essential for effectively supporting the training of DL models for edge computing systems, where the available resources are limited.…”

Section: A Reference Frameworkmentioning

confidence: 99%

See 1 more Smart Citation

A Stochastic Approach for Scheduling AI Training Jobs in GPU-Based Systems

Filippini,

Anselmi,

Ardagna

et al. 2024

IEEE Trans. Cloud Comput.

View full text Add to dashboard Cite

In this work, we optimize the scheduling of Deep Learning (DL) training jobs from the perspective of a Cloud Service Provider running a data center, which efficiently selects resources for the execution of each job to minimize the average energy consumption while satisfying time constraints. To model the problem, we first develop a Mixed-Integer Non-Linear Programming formulation. Unfortunately, the computation of an optimal solution is prohibitively expensive, and to overcome this difficulty, we design a heuristic STochastic Scheduler (STS). Exploiting the probability distribution of early termination, STS determines how to adapt the resource assignment during the execution of the jobs to minimize the expected energy cost while meeting the job due dates. The results of an extensive experimental evaluation show that STS guarantees significantly better results than other methods in the literature, effectively avoiding due date violations and yielding a percentage total cost reduction between 32% and 80% on average. We also prove the applicability of our method in real-world scenarios, as obtaining optimal schedules for systems of up to 100 nodes and 400 concurrent jobs requires less than 5 seconds. Finally, we evaluated the effectiveness of GPU sharing, i.e., running multiple jobs in a single GPU. The obtained results demonstrate that depending on the workload and GPU memory, this further reduces the energy cost by 17-29% on average.

show abstract

Section: Related Workmentioning

confidence: 99%