OverFlow: Multi-Site Aware Big Data Management for Scientific Workflows on Clouds

Abstract

International audienceThe global deployment of cloud datacenters is enabling large scale scientific workflows to improve performance and deliver fast responses. This unprecedented geographical distribution of the computation is doubled by an increase in the scale of the data handled by such applications, bringing new challenges related to the efficient data management across sites. High throughput, low latencies or cost-related trade-offs are just a few concerns for both cloud providers and users when it comes to handling data across datacenters. Existing solutions are limited to cloud-provided storage, which offers low performance based on rigid cost schemes. In turn, workflow engines need to improvise substitutes, achieving performance at the cost of complex system configurations, maintenance overheads, reduced reliability and reusability. In this paper, we introduce OverFlow, a uniform data management system for scientific workflows running across geographically distributed sites, aiming to reap economic benefits from this geo-diversity. Our solution is environment-aware, as it monitors and models the global cloud infrastructure, offering high and predictable data handling performance for transfer cost and time, within and across sites. OverFlow proposes a set of pluggable services, grouped in a data scientist cloud kit. They provide the applications with the possibility to monitor the underlying infrastructure, to exploit smart data compression, deduplication and geo-replication, to evaluate data management costs, to set a tradeoff between money and time, and optimize the transfer strategy accordingly. The system was validated on the Microsoft Azure cloud across its 6 EU and US datacenters. The experiments were conducted on hundreds of nodes using synthetic benchmarks and real-life bio-informatics applications (A-Brain, BLAST). The results show that our system is able to model accurately the cloud performance and to leverage this for efficient data dissemination, being able to reduce the monetary costs and transfer time by up to 3 times

Development of a method for selecting the approximatimg functions for the observable processes of cloud infrastructure

“…Various methods of technical and organizational nature were developed to solve this problem: load redistribution by the communication channels [8,9], compression of transmitted data [10], statistical processing of measurements [11] and others [12]. However, there is a not solved issue of ensuring the required accuracy of measurements when achieving the goal of reducing the total computational burden.…”

A Cache-Based Data Movement Infrastructure for On-demand Scientific Cloud Computing

“…Frequently, a central storage site keeps long-term use data for pipelines. The common data access patterns of these pipelines include data dissemination, collection, and aggregation [37]. In addition, concurrent data write operations across different phases are very rare.…”

“…Accordingly, accelerating data analysis for each stage may require computing facilities that are located in different clouds. Between different stages of the geographical data pipeline, moving a large amount of data across clouds is common [5,37]. This type of multi-cloud environment can consist of resources from multiple public cloud vendors, such as Amazon AWS [1] and Microsoft Azure [29], and private data centers.…”

“…Recent projects support directly transferring files between sites to improve overall system efficiency [38]. For example, OverFlow [37,39] provides a uniform storage management system for multi-site workflows that utilize the local disks associated with virtual machine instances. It extends replication service to handle data transfer for intersite and intra-site traffic using different protocols and mechanisms.…”

Abstract Intelligence

“…This brings us to the basic concept of Cloudets (Baciu et al, 2012). The ability to monitor and automatically redeploy resources (Baciu et al, 2015), within a cloud orchestration framework, aided by machine learning for predicting resource utilization and load rebalancing (Tudoran et al, 2016) makes the Cloudet architecture ideal for exploring the cloud vs brain metaphor in the context of the global Internet. (This section is contributed by Prof. George Baciu.…”