A Genetic Algorithm Based Data Replica Placement Strategy for Scientific Applications in Clouds

Abstract

Cloud computing is a promising distributed computing platform for big data applications, e.g., scientific applications, since excessive resources can be obtained from cloud services for processing and storing both existing and generated application datasets. However, when tasks process big data stored in distributed data centers, the inevitable data movements will cause huge bandwidth cost and execution delay. In this paper, we construct a tripartite graph based model to formulate the data replica placement problem and propose a genetic algorithm based data replica placement strategy for scientific applications to reduce data transmissions in cloud. Our approach can reduce 1) the size of moved data, 2) the time of data movement and 3) the number of movements. We conduct experiments to compare the proposed strategy with the random placement strategy used in Hadoop Distributed Files System (HDFS), which demonstrates that our strategy has better performance for scientific applications in clouds.

Cost-Driven Off-Loading for DNN-Based Applications Over Cloud, Edge, and End Devices

“…A DNN and a scientific workflow have many similarities, such as the overall structure, and the data dependencies between each pair of computing nodes. Cui et al [18] proposed a data placement strategy based on GA for a scientific workflow to reduce the amount of data movement in cloud environment. They modified the mutation and crossover operator of GA to get a good performance from a global perspective.…”

“…5) Parameter Settings: The inertia weight w affects the convergence and search ability of PSO [29]. Formula (18) represents an adjustment mechanism for the inertia weight [30] w = w max − iters cur × w max − w min iters max (18) where w min and w max are the given minimum and maximum of w in the initialization phase. iters cur and iters max are the current number and the maximum number of iterations, respectively.…”

“…To verify the effectiveness of PSO-GA, we modified the GA [18] and the Greedy [23] to adapt the cost-driven off-loading strategy for DNNs in hybrid computing environments.…”

Data privacy‐based coordinated placement method of workflows and data

“…Based on the overall placement of meteorological historical data, we continue to study how to collaboratively place the input data and tasks of workflows on the corresponding nodes in cloud centre [8,10], thereby further reducing the average data access time for all tasks in workflows [11,12]. As the number of meteorological workflows and data offloaded to cloud centre increases rapidly [13,14], increasing the resource utilisation of active nodes in cloud centre is also being paid more and more attention [15,16], and it has become an important indicator to measure the performance of placement method [17,18]. In addition, with the improvement of the confidentiality of meteorological data, there are some privacy conflicts between meteorological data, so that these conflicting meteorological data should avoid being placed on the same or neighbouring storage nodes to ensure the security of these privacy data [19][20][21].…”

Fuzzy Theory-Based Data Placement for Scientific Workflows in Hybrid Cloud Environments

“…However, they did not consider some essential factors in data placement, such as differences in data centers (e.g., capacities and bandwidth) and bandwidth fluctuations. Zheng et al [17] and Cui et al [18] developed a data placement scheme based on the GA, which may easily fall into the local optimal solution during operation. As for optimization objectives in data placement, Liu et al [19] set the transmission times of crossing data centers as an objective, Deng et al [20] and Zhao et al [21] targeted the data transmission volume, and Chen et al [22] aimed for reducing the transmission costs.…”