Towards Data Intensive Many-Task Computing

Abstract

Many-task computing aims to bridge the gap between two computing paradigms, high throughput computing and high performance computing. Traditional techniques to support many-task computing commonly found in scientific computing (i.e. the reliance on parallel file systems with static configurations) do not scale to today’s largest systems for data intensive application, as the rate of increase in the number of processors per system is outgrowing the rate of performance increase of parallel file systems. In this chapter, the authors argue that in such circumstances, data locality is critical to the successful and efficient use of large distributed systems for data-intensive applications. They propose a “data diffusion” approach to enable data-intensive many-task computing. They define an abstract model for data diffusion, define and implement scheduling policies with heuristics that optimize real world performance, and develop a competitive online caching eviction policy. They also offer many empirical experiments to explore the benefits of data diffusion, both under static and dynamic resource provisioning, demonstrating approaches that improve both performance and scalability.

HyCache: A User-Level Caching Middleware for Distributed File Systems

“…FusionFS has already scaled to 1K nodes, and we aim to scale up FusionFS+HyCache to 10K nodes. We will also apply HyCache to Many-Task Computing (MTC) [31][32][33][34], which has specific emphasis on data-intensive computing [35] and cloud computing [36].…”

ZHT: A Light-Weight Reliable Persistent Dynamic Scalable Zero-Hop Distributed Hash Table

“…FusionFS is optimized for a subset of HPC and many-task computing (MTC) [12,59,62,63] workloads, and it is designed for extreme scales [61]. These workloads are often extremely data-intensive [56,58,60], and optimizing data locality [55] becomes critical to achieving good scalability and performance.…”

Modeling Many-Task Computing Workloads on a Petaflop IBM Blue Gene/P Supercomputer

“…Due to the independence of the tasks, BOT applications have been considered most suitably to be executed over widely distributed computational grids. However, we believe that BOT applications should also be able to be executed on other large-scale systems, such as clouds with the help of workflow systems and the ManyTask Computing (MTC) paradigm [7] [45].…”

“…Tasks can be grouped into BOT within one layer, and be scheduled together in the unit of BOT. A large number of applications [42][44] [45] have been covered with this new programming model, spanning everything from supercomputers [44] to grids [50] and clouds [47], and dataintensive systems [46]. Though there are differences between the MTC-based scientific computing workloads and the initial target workloads of clouds (e.g.…”