Data placement for scientific applications in distributed environments

Chervenak, Ann; Deelman, Ewa; Livny, Miron; Su, Min; Schuler, Robert; Bharathi, S.; Mehta, Gaurang; Vahi, Karan

doi:10.1109/grid.2007.4354142

Cited by 86 publications

(49 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Much research interest is concentrated on workflow performance improvement and such strategies as task clustering [5], task/data throttling [13] and data staging [1] are the mostly used techniques to address the issues of performance optimization. In order to improve performance of fine co mputational granularity task scientific workflows, tas k clustering [5] can minimize the completion time of the workflow by reducing the impact of queue waiting t ime.…”

Section: Relat Ed Workmentioning

confidence: 99%

Workflow System Performance Improving and Analyzing by Using Memory Cache

Xu¹,

An²,

Xu³

2015

Proceedings of the 2015 International Conference on Computer Science and Intelligent Communication

View full text Add to dashboard Cite

Abstract-Workflow techniques may be broadly used in business processing and scientific computing and thus focused research interest over the last decade. The performance of workflow applications mainly depend upon task scheduling strategies in workflow engine. In this paper, a novel active workflow instance memory cache (WfIC) architecture is proposed, which is used for improving workflow system performance. By caching active workflow instances in main memory, instance database IO operations are pruned during task scheduling and processing. Additionally, theory analysis shows that WfIC can effectively improve the performance of workflow systems. Also, theory analysis gives an instruction on how to configure the instance cache and the lower/upper limit of the instance executing time.

show abstract

Section: Relat Ed Workmentioning

confidence: 99%

Workflow System Performance Improving and Analyzing by Using Memory Cache

Xu¹,

An²,

Xu³

2015

Proceedings of the 2015 International Conference on Computer Science and Intelligent Communication

View full text Add to dashboard Cite

show abstract

“…A different approach is followed by [11]. Its authors, like those of [4], argue that in order to improve efficiency and also simplify the systems, it is best to separate the data placement and the scheduling machinery.…”

Section: Related Workmentioning

confidence: 99%

“…It is out of the scope of the present work to develop a component that completely solve these challenges but, in the line of [11], we believe that it would be best to decouple job scheduling from data replication tasks. Theses are best handled by existing data placement systems.…”

Section: Proposal For An Enhanced Solutionmentioning

confidence: 99%

Data location-aware job scheduling in the grid. Application to the GridWay metascheduler

Peris

Hernández

Huedo

et al. 2010

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

Abstract. Grid infrastructures constitute nowadays the core of the computing facilities of the biggest LHC experiments. These experiments produce and manage petabytes of data per year and run thousands of computing jobs every day to process that data. It is the duty of metaschedulers to allocate the tasks to the most appropriate resources at the proper time.Our work reviews the policies that have been proposed for the scheduling of grid jobs in the context of very data-intensive applications. We indicate some of the practical problems that such models will face and describe what we consider essential characteristics of an optimum scheduling system: aim to minimise not only job turnaround time but also data replication, flexibility to support different virtual organisation requirements and capability to coordinate the tasks of data placement and job allocation while keeping their execution decoupled.These ideas have guided the development of an enhanced prototype for GridWay, a general purpose metascheduler, part of the Globus Toolkit and member of the EGEE's RESPECT program. Current GridWay's scheduling algorithm is unaware of data location. Our prototype makes it possible for job requests to set data needs not only as absolute requirements but also as functions for resource ranking. As our tests show, this makes it more flexible than currently used resource brokers to implement different data-aware scheduling algorithms.

show abstract

“…They range from storage glide-ins (e.g., BADFS [10]) to building application-optimized storage systems (e.g. HDFS [11], BADFS [10]), to building a configurable storage system that is tuned at deployment time to better support a specific application [12], to offering specific data access optimizations (e.g., location-aware scheduling [13], caching, and data placement techniques [14]). Taken in isolation, these efforts do not fully address the problem we face as they are either specific to a class of applications (e.g., HDFS for map-reduce applications), and consequently incapable to support a large set of workflow applications; or enable system-wide optimizations throughout the application runtime, thus inefficiently supporting applications that have different usage patterns for different files.…”

Section: Introductionmentioning

confidence: 99%

A Workflow-Aware Storage System: An Opportunity Study

Vairavanathan

Al-Kiswany

Costa

et al. 2012

2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (Ccgrid 2012)

View full text Add to dashboard Cite

-This paper evaluates the potential gains a workflow-aware storage system can bring. Two observations make us believe such storage system is crucial to efficiently support workflow-based applications: First, workflows generate irregular and application-dependent data access patterns. These patterns render existing storage systems unable to harness all optimization opportunities as this often requires conflicting optimization options or even conflicting design decision at the level of the storage system. Second, when scheduling, workflow runtime engines make suboptimal decisions as they lack detailed data location information. This paper discusses the feasibility, and evaluates the potential performance benefits brought by, building a workflow-aware storage system that supports per-file access optimizations and exposes data location. To this end, this paper presents approaches to determine the application-specific data access patterns, and evaluates experimentally the performance gains of a workflowaware storage approach. Our evaluation using synthetic benchmarks shows that a workflow-aware storage system can bring significant performance gains: up to 7x performance gain compared to the distributed storage system -MosaStore and up to 16x compared to a central, well provisioned, NFS server.

show abstract

Data placement for scientific applications in distributed environments

Cited by 86 publications

References 20 publications

Workflow System Performance Improving and Analyzing by Using Memory Cache

Workflow System Performance Improving and Analyzing by Using Memory Cache

Data location-aware job scheduling in the grid. Application to the GridWay metascheduler

A Workflow-Aware Storage System: An Opportunity Study

Contact Info

Product

Resources

About