Optimal Load Balancing in Bipartite Graphs

Abstract

Applications in cloud platforms motivate the study of efficient load balancing under job-server constraints and server heterogeneity. In this paper, we study load balancing on a bipartite graph where left nodes correspond to job types and right nodes correspond to servers, with each edge indicating that a job type can be served by a server. Thus edges represent locality constraints, i.e., each job can only be served at servers which contained certain data and/or machine learning (ML) models. Servers in this system can have heterogeneous service rates. In this setting, we investigate the performance of two policies named Join-the-Fastest-of-the-Shortest-Queue (JFSQ) and Join-the-Fastest-of-the-Idle-Queue (JFIQ), which are simple variants of Join-the-Shortest-Queue and Jointhe-Idle-Queue, where ties are broken in favor of the fastest servers. Under a "well-connected" graph condition, we show that JFSQ and JFIQ are asymptotically optimal in the mean response time when the number of servers goes to infinity. In addition to asymptotic optimality, we also obtain upper bounds on the mean response time for finite-size systems. We further show that the wellconnectedness condition can be satisfied by a random bipartite graph construction with relatively sparse connectivity. However, the above classical load balancing model may not be appropriate for certain modern cloud computing and data analytic applications due to the presence of job-server constraints. Under such constraints, a job can only be dispatched to a subset of the N servers. These constraints, often called locality constraints, are quite common in large-scale Machine Learning as a Service (MLaaS) and serverless computing services supported by cloud computing

Load Balancing in Heterogeneous Server Clusters: Insights From a Product-Form Queueing Model

“…The main difficulty is that their expressions involve fractions of two sums that both span all states. We would also like to generalize these results in other directions, for instance by considering open variants of the algorithm or by accounting for assignment constraints [7,21].…”

“…To achieve good performance without these strong assumptions, more recent works introduced speed-aware variants of the above-mentioned well-known algorithms. More specifically, [21] introduced variants of join-the-shortest-queue and join-idle-queue where the server speeds are used as a tie-breaking rule, and proved that these variants minimize the mean response time in the many-server regime; [10] proposed variants of power-of-d-choices and join-idle-queue for service systems with two server types (fast and slow) by adapting the degree of diversity and assignment probabilities to the server speeds, and proved stability, again in the many-server regime. Despite these advances, we still lack a fundamental understanding of the impact of heterogeneity on performance in service systems with a finite number of servers.…”

Theoretically Guaranteed Online Workload Dispatching for Deadline-Aware Multi-Server Jobs

“…RELATED WORKS Online load balancing policies are fully investigated under classic settings, where multiple identical servers with exponentially distributed service rates process continuous arrived jobs. Based on CTMC and Lyapunov Stability theories, load balancing policies such as JSQ [14], JIQ [11], Pod [12], and JFIQ [9] are proposed and analyzed on the mean response time and cross-server communication overhead. In a most recent work [9], Weng et al proposed the JFSQ and JFIQ policies under the constraints of heterogenous service rates and service locality.…”

“…Based on CTMC and Lyapunov Stability theories, load balancing policies such as JSQ [14], JIQ [11], Pod [12], and JFIQ [9] are proposed and analyzed on the mean response time and cross-server communication overhead. In a most recent work [9], Weng et al proposed the JFSQ and JFIQ policies under the constraints of heterogenous service rates and service locality. They show that, under a well-connected bipartite graph condition, these two policies can achieve the minimum mean response time in both the many-server regime and the sub Halfin-Whitt regime.…”

“…Meanwhile, a series of academic studies on online load balancing, which promise long-term performance guarantees, assume that jobs arrive according to Poisson process and service rates of computing instances are exponentially distributed [9]- [13]. Under stochastic ordering assumption, policies such as Join-the-Shortest-Queue (JSQ) [14], Join-the-Idle-Queue (JIQ) [11], Power-of-d-Choices (Pod) [12], and Jointhe-Fastest-of-the-Shortest-Queues (JFIQ) [9] are raised and analyzed based on Continuous-Time Markov Chains (CTMC) and Lyapunov Stability theories. However, their performance guarantees (mostly on mean response time) are established on sufficient assumptions, which are tough to satisfy in production systems.…”