A general architecture for load balancing in a distributed-memory environment

Abstract

The goal of load balancing is to assign to each node a number of tasks proportional to its performance. On distributed-memory machines, it is important to take data dependencies into account when distributing tasks, since they have a big impact on the communication requirements of the distributed application. Many load balancers have been proposed that deal with applications with homogeneous tasks, but applications with heterogeneous tasks have proven to be far more complex to handle. In this paper we present a load balancing architecture that can deal with applications with heterogeneous tasks. The idea is to provide a set of load balancers that are effective for different types of homogeneous tasks, and to allow users to combine these load balancers for applications with heterogeneous tasks. We implemented this architecture on the Nectar multicomputer and we present performance results for several applications with homogeneous and heterogeneous tasks.

Network-based multicomputers: a practical supercomputer architecture

“…As described earlier in this section, programming tools for both tightly-coupled and loosely-coupled distributed-memory systems are an active area of research. In the context of the Nectar system, we have demonstrated tools in three critical areas: monitoring tools that help the programmer understand the behavior of the application [47,12], support for data sharing across the network [48,49], and load balancing tools that help in distributing work to make efficient use of the cycles on the nodes [67,66,67,53,54,55].…”

Customized Dynamic Load Balancing for a Network of Workstations

“…The task queue model for dynamic loop scheduling has targeted shared-memory machines [11,14], while the diffusion model has been used for distributedmemory machines [10]. A method for task-level scheduling in heterogeneous programs was proposed in [13], and [2] presents an application-specific approach to schedule individual parallel applications.…”

Gigabit Nectar: Architecture and Performance.

“…Implementing applications on a network-based multicomputer is a non-trivial effort, and programming tools that simplify that task are needed. In the context of the Nectar and Gigabit Nectar systems, we have demonstrated tools in three critical areas: monitoring tools that help the programmer understand the behavior of their application [10,9], support for data sharing across the network [35,36], and load balancing tools that help in distributing work to make efficient use of the cycles on the nodes [51,50,41,42,40].…”