A Model-Based Approach to Streamlining Distributed Training for Asynchronous SGD

Lin, Sung-Han; Paolieri, Marco; Chou, Cheng-Fu; Golubchik, Leana

doi:10.1109/mascots.2018.00037

Cited by 25 publications

(30 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The closest work to our results is [29], which considers jobs which follow a realistic speedup function and have known sizes. [29] also allows server allocations to change over time.…”

Section: Prior Workmentioning

confidence: 99%

See 1 more Smart Citation

heSRPT: Parallel scheduling to minimize mean slowdown

Berg

Vesilo

2020

Performance Evaluation

View full text Add to dashboard Cite

Modern data centers serve workloads which are capable of exploiting parallelism. When a job parallelizes across multiple servers it will complete more quickly, but jobs receive diminishing returns from being allocated additional servers. Because allocating multiple servers to a single job is inefficient, it is unclear how best to allocate a fixed number of servers between many parallelizable jobs. This paper provides the first optimal allocation policy for minimizing the mean slowdown of parallelizable jobs of known size when all jobs are present at time 0. Our policy provides a simple closed form formula for the optimal allocations at every moment in time. Minimizing mean slowdown usually requires favoring short jobs over long ones (as in the SRPT policy). However, because parallelizable jobs have sublinear speedup functions, system efficiency is also an issue. System efficiency is maximized by giving equal allocations to all jobs and thus competes with the goal of prioritizing small jobs. Our optimal policy, high-efficiency SRPT (heSRPT), balances these competing goals. heSRPT completes jobs according to their size order, but maintains overall system efficiency by allocating some servers to each job at every moment in time. Our results generalize to also provide the optimal allocation policy with respect to mean flow time. Finally, we consider the online case where jobs arrive to the system over time. While optimizing mean slowdown in the online setting is even more difficult, we find that heSRPT provides an excellent heuristic policy for the online setting. In fact, our simulations show that heSRPT significantly outperforms state-of-the-art allocation policies for parallelizable jobs.

show abstract

“…The closest work to our results is [29], which considers jobs which follow a realistic speedup function and have known sizes. [29] also allows server allocations to change over time.…”

Section: Prior Workmentioning

confidence: 99%

“…The closest work to our results is [29], which considers jobs which follow a realistic speedup function and have known sizes. [29] also allows server allocations to change over time. [29] proposes and evaluates heuristic policies such as HELL and KNEE, but they make no theoretical guarantee about the performance of their policies.…”

Section: Prior Workmentioning

confidence: 99%

heSRPT: Parallel scheduling to minimize mean slowdown

Berg

Vesilo

2020

Performance Evaluation

View full text Add to dashboard Cite

show abstract

“…In particular, the jobs typically request multiple servers simultaneously and hold onto them for the duration of the job. This difference is largely a result of machine learning jobs like TensorFlow [6], which are highly parallel. For example, when we look at Google's Borg Scheduler [10], we see that the number of servers occupied by an individual job can be anywhere from 1 to 100000 [13].…”

Section: Introductionmentioning

confidence: 99%

Zero Queueing for Multi-Server Jobs

Wang

Xie

2021

Abstract Proceedings of the 2021 ACM SIGMETRICS / International Conference on Measurement and Modeling of Computer Systems

View full text Add to dashboard Cite

Cloud computing today is dominated by multi-server jobs. These are jobs that request multiple servers simultaneously and hold onto all of these servers for the duration of the job. Multi-server jobs add a lot of complexity to the traditional one-server-per-job model: an arrival might not "fit" into the available servers and might have to queue, blocking later arrivals and leaving servers idle. From a queueing perspective, almost nothing is understood about multiserver job queueing systems; even understanding the exact stability region is a very hard problem.In this paper, we investigate a multi-server job queueing model under scaling regimes where the number of servers in the system grows. Specifically, we consider a system with multiple classes of jobs, where jobs from different classes can request different numbers of servers and have different service time distributions, and jobs are served in first-come-first-served order. The multi-server job model opens up new scaling regimes where both the number of servers that a job needs and the system load scale with the total number of servers. Within these scaling regimes, we derive the first results on stability, queueing probability, and the transient analysis of the number of jobs in the system for each class. In particular we derive sufficient conditions for zero queueing. Our analysis introduces a novel way of extracting information from the Lyapunov drift, which can be applicable to a broader scope of problems in queueing systems.

show abstract

“…Prior Work The sub-linear-sched problem has been an object of immense interest [3,4,11,7,8,1], where practical algorithms include packing based [18], and resource reservation algorithms [14]. Heuristic policies with only numerical performance analysis can be found in [13]. In past, this problem has been considered for the combinatorial discrete allocation model [11], where an integer number of servers are assigned to any job, as well as the continuous allocation model [7,8,1,3,4], that treats the N servers as a single resource block which can be partitioned into any size and assigned to any job.…”

Section: Introductionmentioning

confidence: 99%

Speed Scaling with Multiple Servers Under A Sum Power Constraint

Vaze¹,

Nair²

2021

Preprint

View full text Add to dashboard Cite

The problem of scheduling jobs and choosing their respective speeds with multiple servers under a sum power constraint to minimize the flow time + energy is considered. This problem is a generalization of the flow time minimization problem with multiple unit-speed servers, when jobs can be parallelized, however, with a sub-linear, concave speedup function k 1/α , α > 1 when allocated k servers, i.e., jobs experience diminishing returns from being allocated additional servers. When all jobs are available at time 0, we show that a very simple algorithm EQUI, that processes all available jobs at the same speed is 2-competitive, while in the general case, when jobs arrive over time, an LCFS based algorithm is shown to have a constant (dependent only on α) competitive ratio. * We acknowledge support of the Department of Atomic Energy, Government of India, under project no. RTI4001

show abstract

A Model-Based Approach to Streamlining Distributed Training for Asynchronous SGD

Cited by 25 publications

References 21 publications

heSRPT: Parallel scheduling to minimize mean slowdown

heSRPT: Parallel scheduling to minimize mean slowdown

Zero Queueing for Multi-Server Jobs

Speed Scaling with Multiple Servers Under A Sum Power Constraint

Contact Info

Product

Resources

About