Cost-Efficient High-Performance Internet-Scale Data Analytics over Multi-cloud Environments

Abstract

Abstract-To analyze data distributed across the world, one can use distributed computing power to take advantage of data locality and achieve higher throughput. The multi-cloud model, a composition of multiple clouds, can provide cost-effective computing resources to process such distributed data. As multicloud becomes more and more accessible from cloud users, the use of MapReduce/Hadoop over multi-cloud is emerging; however, existing work has two issues in principle. First, it mainly focuses on maximizing throughput by improving data locality, but the perspective of cost optimization is missing. Second, conventional centralized optimization methods would not be able to scale well in multi-cloud environments due to its highly dynamic nature. We plan to solve the first issue by formalizing an optimization framework for MapReduce over multi-cloud including virtual machine and data transfer costs, and then the second issue by creating decentralized resource management middleware that considers multi-criteria (cost and performance) optimization. This paper reports progress we have made so far on these two directions.

Multi-Cloud Query Optimisation with Accurate and Efficient Quoting

“…The idea of using the readily available computing capabilities of cloud providers hosting public data for distributed analytics originally arose for MapReduce. Although not directly applicable to relational databases available through DBaaS, it has nevertheless been shown that using multi-cloud resources for processing is cost-effective -in the sense that it can be less expensive than single-cloud or local processing [2].…”

“…The topic of multi-cloud data management raised interests in the industry as well as in the DB research community (as highlighted by the Seattle Report [1]). Specialised middlewares for multi-cloud database integration have consequently been introduced [2], [3] to let users write multi-cloud queries independently from provider-arranged multi-cloud federations. In previous work [3], we identified two main design requirements for such a middleware.…”

Towards Multi-Objective Optimisation of Hadoop 2.x Application Deployment on Public Clouds

“…(1) Provide requirements (e.g., budget constraints) [14] tackles the optimisation problem without taking into account the cost of I/O operations, which significantly affect on the cost of cloud-deployed Hadoop applications. Similarly, the performance model introduced by Lin et al [15] to predict the performance of MapReduce tasks does not consider the competition for resources between concurrent map and reduce tasks.…”

Resource Scheduling in Cloud Environmet: A Survey

“…In [31], authors formalized an optimization framework for MapReduce over multi-cloud including virtual machine and data transfer costs. In addition, a decentralized resource management middleware that considers multi-optimization is designed.…”