On-demand Overlay Networks for Large Scientific Data Transfers

Ramakrishnan, Lavanya; Guok, Chin; Jackson, Keith; Kissel, Ezra; Swany, Martin; Agarwal, D.

doi:10.1109/ccgrid.2010.82

Cited by 12 publications

(9 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The ESnet deployment and OSCARS integration are described in [9]. The dynamic circuit, buffer and burst implementation and results are described in [5].…”

Section: Phoebus Results and Findingsmentioning

confidence: 99%

Phoebus: Network Middleware for Next-Generation Network Computing

Swany¹

2012

View full text Add to dashboard Cite

“…The ESnet deployment and OSCARS integration are described in [9]. The dynamic circuit, buffer and burst implementation and results are described in [5].…”

Section: Phoebus Results and Findingsmentioning

confidence: 99%

Phoebus: Network Middleware for Next-Generation Network Computing

Swany¹

2012

View full text Add to dashboard Cite

“…A-Brain execution times across 3 sites, using Azure Blobs and OverFlow as backends, to transfer data towards the NUS datacenter high, sustained transfer rates. Similarly, [26] and [27] considered the problem of scheduling data-intensive workflows in clouds assuming that files are replicated in multiple execution sites. These approaches can reduce the makespan of the workflows but come at the cost and overhead of replication.…”

Section: Related Workmentioning

confidence: 99%

OverFlow: Multi-Site Aware Big Data Management for Scientific Workflows on Clouds

Tudoran

Costan²,

Antoniu³

2016

IEEE Trans. Cloud Comput.

View full text Add to dashboard Cite

International audienceThe global deployment of cloud datacenters is enabling large scale scientific workflows to improve performance and deliver fast responses. This unprecedented geographical distribution of the computation is doubled by an increase in the scale of the data handled by such applications, bringing new challenges related to the efficient data management across sites. High throughput, low latencies or cost-related trade-offs are just a few concerns for both cloud providers and users when it comes to handling data across datacenters. Existing solutions are limited to cloud-provided storage, which offers low performance based on rigid cost schemes. In turn, workflow engines need to improvise substitutes, achieving performance at the cost of complex system configurations, maintenance overheads, reduced reliability and reusability. In this paper, we introduce OverFlow, a uniform data management system for scientific workflows running across geographically distributed sites, aiming to reap economic benefits from this geo-diversity. Our solution is environment-aware, as it monitors and models the global cloud infrastructure, offering high and predictable data handling performance for transfer cost and time, within and across sites. OverFlow proposes a set of pluggable services, grouped in a data scientist cloud kit. They provide the applications with the possibility to monitor the underlying infrastructure, to exploit smart data compression, deduplication and geo-replication, to evaluate data management costs, to set a tradeoff between money and time, and optimize the transfer strategy accordingly. The system was validated on the Microsoft Azure cloud across its 6 EU and US datacenters. The experiments were conducted on hundreds of nodes using synthetic benchmarks and real-life bio-informatics applications (A-Brain, BLAST). The results show that our system is able to model accurately the cloud performance and to leverage this for efficient data dissemination, being able to reduce the monetary costs and transfer time by up to 3 times

show abstract

“…Cloud based services, like Amazon's CloudFront [18], use the geographical distribution of data to reduce latencies of data transfers. Similarly, [19] and [20] considered the problem of scheduling data-intensive workflows in clouds assuming that files are replicated in multiple execution sites. These approaches can reduce the makespan of the workflows but come at the cost and overhead of replication, which is considerable for large datasets.…”

Section: B Existing Data Management Solutionsmentioning

confidence: 99%

Adaptive file management for scientific workflows on the Azure cloud

Tudoran

Costan

Rad

et al. 2013

2013 IEEE International Conference on Big Data

View full text Add to dashboard Cite

Scientific workflows typically communicate data between tasks using files. Currently, on public clouds, this is achieved by using the cloud storage services, which are unable to exploit the workflow semantics and are subject to low throughput and high latencies. To overcome these limitations, we propose an alternative leveraging data locality through direct file transfers between the compute nodes. We rely on the observation that workflows generate a set of common data access patterns that our solution exploits in conjunction with context information to self-adapt, choose the most adequate transfer protocol and expose the data layout within the virtual machines to the workflow engines. This file management system was integrated within the Microsoft Generic Worker workflow engine and was validated using synthetic benchmarks and a real-life application on the Azure cloud. The results show it can bring significant performance gains: up to 5x file transfer speedup compared to solutions based on standard cloud storage and over 25% application timespan reduction compared to Hadoop on Azure.

show abstract

On-demand Overlay Networks for Large Scientific Data Transfers

Cited by 12 publications

References 24 publications

Phoebus: Network Middleware for Next-Generation Network Computing

Phoebus: Network Middleware for Next-Generation Network Computing

OverFlow: Multi-Site Aware Big Data Management for Scientific Workflows on Clouds

Adaptive file management for scientific workflows on the Azure cloud

Contact Info

Product

Resources

About