An Algorithm for Cost-Effectively Storing Scientific Datasets with Multiple Service Providers in the Cloud

Abstract

Cloud computing provides scientists a platform that can deploy computation and data intensive applications without infrastructure investment. With excessive cloud resources and a decision support system, large generated datasets can be flexibly 1) stored locally in the current cloud, 2) deleted and regenerated whenever reused or 3) transferred to cheaper cloud service for storage. However, due to the pay-as-you-go model, the total application cost largely depends on the usage of computation, storage and bandwidth resources, hence cutting the cost of cloud-based data storage becomes a big concern for deploying scientific applications in the cloud. In this paper, we propose a novel strategy that can costeffectively store large generated datasets with multiple cloud service providers. The strategy is based on a novel algorithm that finds the trade-off among computation, storage and bandwidth costs in the cloud, which are three key factors for the cost of data storage. Both general (random) simulations conducted with popular cloud service providers' pricing models and three specific case studies on real world scientific applications show that the proposed storage strategy is highly cost effective and practical for runtime utilisation in the cloud. 1 This paper is a significantly extended version of our conference paper [32]. The extensions are from the following aspects: 1) more comprehensive description of the T-CSB algorithm, including the new theorem, figure and algorithm pseudo code; 2) utilisation of the T-CSB algorithm to a localoptimisation based cost-effective and efficient storage strategy; 3) efficiency evaluation of the proposed strategy; 4) case studies of real scientific applications of using the proposed strategy.The remainder of this paper is organised as follows. Section 2 presents a motivating example of scientific application and analyses the research problems. Section 3 introduces some preliminaries and data storage cost model in the cloud. Section 4 presents our novel cost-effective storage strategy in detail. Section 5 describes our experimental results for evaluation. Section 6 discusses the related work. Section 7 summarises our conclusions and points out future work.

Cloud Data Management for Scientific Workflows: Research Issues, Methodologies, and State-of-the-Art

“…Based on the optimised placement of data, effective and efficient task scheduling approaches can be derived [62] [63] [64] [64]. Furthermore, as numerous cloud services have been published on Internet, in order to reduce the data storage cost, less frequently used data can be moved to cheap cloud services for storage [65] [66].…”

A Genetic Algorithm Based Data Replica Placement Strategy for Scientific Applications in Clouds

“…In real world scientific applications (e.g., Montage [30], SIPHT [31], Pulsar Searching [32] [33], FEM [34] and Epigenomics [35], etc. ), the size of datasets varies from tens of megabytes to tens of gigabytes, and number of tasks varies from tens to thousands.…”

Multi-provider cloud computing network infrastructure optimization

“…1, an enterprise-level cloud cache can be deployed in a multi-provider cloud environment where a consumer enterprise employs multiple concurrent cloud providers to host its external private clouds that are either independent from each other [13,14] or interconnected as a hybrid cloud [15]; both are the forms of federated cloud [16,17]. There are several benefits of multi-provider cloud deployment such as load balancing [18,19], enabling of planned downtime for system maintenance [17], confidentiality protection [20], risk mitigation (mandated by hospitals, stock markets, air transportation controls, etc.) [20,21] and the utilization of unique capabilities offered by different cloud providers [13].…”