Scrooge

Hu, Yixuan; Ghosh, Rajrup; Govindan, Ramesh

doi:10.1145/3472883.3486993

Cited by 20 publications

(6 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Among these, note that GPU sharing is currently gaining remarkable interest, due to the large capacity of the most recent hardware. In particular, a time sharing [18] or space sharing [19] pattern can be considered, related to the fact that jobs either exploit the entire GPU for a limited time fraction or spatially share the GPU resources with a limited percentage. While we focus on space sharing in our work, both strategies are considered in this section.…”

Section: Related Workmentioning

confidence: 99%

“…Scrooge [19] proposes a Mixed-Integer Linear Programming formulation to find the minimum-cost GPU-accelerated Virtual Machine in the cloud, meeting performance objectives. Similarly to our work, the problem is solved whenever a new job leaves or joins.…”

Section: Related Workmentioning

confidence: 99%

“…Note that, while the interference generated by node partitioning can be neglected (see, e.g., [11]- [15]), this is not generally true for GPU sharing. Nevertheless, we decided to consider this scenario, which is currently gaining popularity in the literature [19], [22], since the large memory of the most recent GPUs usually allows to simultaneously execute multiple mini-jobs, freeing resources for more demanding jobs. Moreover, GPU sharing is essential for effectively supporting the training of DL models for edge computing systems, where the available resources are limited.…”

Section: A Reference Frameworkmentioning

confidence: 99%

“…where the first parenthesis corresponds to the total execution time of job j with the current configuration (defined, as in Constraints (19), by summing the contribution obtained if j is deployed on a GPU fraction and the one obtained if it is deployed on dedicated GPUs). The second parenthesis is the energy cost of node n, since γ nκ is 1 when a total of κ GPUs are used on node n. Note that, with this definition, job j pays the energy cost due to the entire node usage, even if the k GPUs are partitioned among many jobs.…”

Section: Resource Selection-job Scheduling Problemmentioning

confidence: 99%

See 3 more Smart Citations

A Stochastic Approach for Scheduling AI Training Jobs in GPU-Based Systems

Filippini,

Anselmi,

Ardagna

et al. 2024

IEEE Trans. Cloud Comput.

View full text Add to dashboard Cite

In this work, we optimize the scheduling of Deep Learning (DL) training jobs from the perspective of a Cloud Service Provider running a data center, which efficiently selects resources for the execution of each job to minimize the average energy consumption while satisfying time constraints. To model the problem, we first develop a Mixed-Integer Non-Linear Programming formulation. Unfortunately, the computation of an optimal solution is prohibitively expensive, and to overcome this difficulty, we design a heuristic STochastic Scheduler (STS). Exploiting the probability distribution of early termination, STS determines how to adapt the resource assignment during the execution of the jobs to minimize the expected energy cost while meeting the job due dates. The results of an extensive experimental evaluation show that STS guarantees significantly better results than other methods in the literature, effectively avoiding due date violations and yielding a percentage total cost reduction between 32% and 80% on average. We also prove the applicability of our method in real-world scenarios, as obtaining optimal schedules for systems of up to 100 nodes and 400 concurrent jobs requires less than 5 seconds. Finally, we evaluated the effectiveness of GPU sharing, i.e., running multiple jobs in a single GPU. The obtained results demonstrate that depending on the workload and GPU memory, this further reduces the energy cost by 17-29% on average.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: A Reference Frameworkmentioning

confidence: 99%

Section: Resource Selection-job Scheduling Problemmentioning

confidence: 99%

See 2 more Smart Citations

A Stochastic Approach for Scheduling AI Training Jobs in GPU-Based Systems

Filippini,

Anselmi,

Ardagna

et al. 2024

IEEE Trans. Cloud Comput.

View full text Add to dashboard Cite

show abstract

“…Batching and parallelism parameters are practical configuration knobs of ML inference services. Batching refers to aggregating multiple requests into one request, which is widely adapted for GPU inference systems [16,23,33,35]. However, as shown in Figure 4, inference on CPU does not substantially benefit from batching in increasing the throughput, but increasing batch size leads to higher latency.…”

Section: Experimental Evaluationmentioning

confidence: 99%

Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systems

Salmani

Ghafouri

Sanaee

et al. 2023

Proceedings of the 3rd Workshop on Machine Learning and Systems

View full text Add to dashboard Cite

The use of machine learning (ML) inference for various applications is growing drastically. ML inference services engage with users directly, requiring fast and accurate responses. Moreover, these services face dynamic workloads of requests, imposing changes in their computing resources. Failing to right-size computing resources results in either latency service level objectives (SLOs) violations or wasted computing resources. Adapting to dynamic workloads considering all the pillars of accuracy, latency, and resource cost is challenging. In response to these challenges, we propose InfAdapter, which proactively selects a set of ML model variants with their resource allocations to meet latency SLO while maximizing an objective function composed of accuracy and cost. InfAdapter decreases SLO violation and costs up to 65% and 33%, respectively, compared to a popular industry autoscaler (Kubernetes Vertical Pod Autoscaler).

show abstract

DeepLat: Achieving Minimum Worst Case Latency for DNN Inference with Batch-Aware Dispatching

Gao,

2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Scrooge

Cited by 20 publications

References 13 publications

A Stochastic Approach for Scheduling AI Training Jobs in GPU-Based Systems

A Stochastic Approach for Scheduling AI Training Jobs in GPU-Based Systems

Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systems

DeepLat: Achieving Minimum Worst Case Latency for DNN Inference with Batch-Aware Dispatching

Contact Info

Product

Resources

About