A data placement service for petascale applications

Abstract

We examine the use of policy-driven data placement services to improve the performance of data-intensive, petascale applications in high performance distributed computing environments. In particular, we are interested in using an asynchronous data placement service to stage data in and out of application workflows efficiently as well as to distribute and replicate data according to Virtual Organization policies. We propose a data placement service architecture and describe our implementation of one layer of this architecture, which provides efficient, priority-based bulk data transfers.

Automatic data placement and replication in grids

“…Examples include Phedex [23] for CMS, LDR [24] for LIGO (Laser Interferometer Gravitational Wave Observatory) [18] and DRS [7] for the Globus project [1]. These tools rely on users to decide what data to replicate and where to put replicas.…”

“…Many researchers have investigated the data replication problem in grids, such as the work reported in [7,6,25,28,19]. Among the existing work, the most closely related is the research done by Ranganathan and Foster in [20,21,22], where the authors study the relationship between asynchronous data replication and job scheduling.…”

“…First, it generates a list (L h ) that includes the most popular data objects whose replications are expected to cause the workload of site h, to reduce below the target (lines [4][5][6][7][8][9][10][11][12][13][14][15][16]. That is, j∈L h Δλ hj ≥ Δλ h .…”

“…In current data grids, most data replication tools [23,24,7] focus on providing fast data transfer. These tools rely on users to decide where and what data to replicate.…”

A Data Management Framework for Urgent Geoscience Workflows

“…To date, none of these capabilities have addressed urgent computing data storage services. The capabilities of recent services, such as the GridFTP QoS provisioning [16], the Managed Object Placement Service (MOPS) [17], and the Data Placement Service (DPS) [18], provide the means for obtaining and sustaining QoS for a storage resource but cannot manage urgent data requirements without additional support. To adequately support urgent computations and workflows, an additional management layer is required to obtain the QoS best suited for the I/O footprint of an urgent computation, adjust the QoS of other concurrent workflows so that urgent workflows are not starved for resources, and negotiate end-to-end workflow scheduling for all urgent computing resources, such as storage, compute, or network resources through the utilization of available capabilities.…”