SWARM: Scheduling Large-Scale Jobs over the Loosely-Coupled HPC Clusters

Abstract

Abstract-Compute-intensive scientific applications are heavily reliant on the available quantity of computing resources. The Grid paradigm provides a large scale computing environment for scientific users. However, conventional Grid job submission tools do not provide a high-level job scheduling environment for these users across multiple institutions. For extremely large number of jobs, a more scalable job scheduling framework that can leverage highly distributed clusters and supercomputers is required. In this paper, we propose a high-level job scheduling Web service framework, Swarm. Swarm is developed for scientific applications that must submit massive number of high-throughput jobs or workflows to highly distributed computing clusters. The Swarm service itself is designed to be extensible, lightweight, and easily installable on a desktop or small server. As a Web service, derivative services based on Swarm can be straightforwardly integrated with Web portals and science gateways. This paper provides the motivation for this research, the architecture of the Swarm framework, and a performance evaluation of the system prototype.

The Problem Solving Environments of TeraGrid, Science Gateways, and the Intersection of the Two

“…Condor-G in turn uses GRAM2 for its job submission and data staging. We have spun this work off into the Swarm project [31]. Data services in the portal include fault models from the QuakeTables fault database [32], GPS archival data from Geophysical Resources Web Service [33], and real-time data from the California Real-Time Network [34].…”

Log Analysis-Based Resource and Execution Time Improvement in HPC: A Case Study

“…This supercomputer log shows that idle resources are not used during this waiting time, causing inefficient resource waste. Therefore, the efficiency of resource allocation has to be evaluated by the scheduling algorithm [9]. It can be seen that optimization can be performed by applying the backfilling.…”

High-Performance Parallel Computing with Cloud and Cloud Technologies

“…Table 2 shows the data/computation flow of these three basic execution units, along with examples. (Raicu, Zhao et al 2007) and SWARM (Pallickara and Pierce 2008) all provide similar functionality by scheduling large numbers of individual maps/jobs. Applications which can utilize a "reduction" or an "aggregation" operation can use both phases of the MapReduce model and, depending on the "associativity" and "transitivity" nature of the reduction operation, multiple reduction phases can be applied to enhance the parallelism.…”