In-Staging Data Placement for Asynchronous Coupling of Task-Based Scientific Workflows

Exact and Heuristic Data Workflow Placement Algorithms for Big Data Computing in Cloud Datacenters

“…The storage space capacity is considered for the datacenters as randomly set from range [10 GB, 1000 GB] [12]. The transfer link capacity of one unit of intermediate data transmission between distributed datacenters are randomly drawn from range [1,10] Gbps [7] with a random transfer cost (in $) ranging from 0 to 0.09. Both storage and transaction costs (in $) of one unit of intermediate data dependency are set within [0.02, 0.04] and [0, 0.09] respectively, in relation to the typical charges in Amazon S3 3 .…”

“…The research works that considered data workflow features are presented in [10,11,12,13,14]. Nevertheless, the dynamic variation of inter and intra-jobs dependencies from the generated intermediate data was not addressed with the same focus.…”

“…Nevertheless, the dynamic variation of inter and intra-jobs dependencies from the generated intermediate data was not addressed with the same focus. In [10], an adaptive data-task placement approach is proposed that reflects asynchronous coupling among tasks in order to reduce execution time and data movement overhead. The authors in [11] have dealt with improving the data workflow's execution by clustering the interdependent datasets and distribute them intelligently onto the same datacenters to reduce data transfers.…”

“…(t), from the inter-job dependencies, are atomic and must be synchronized for their processing. By contrast, for the intra-job dependencies, these files are deduced from a partial correlation with an asynchronous processing [10]. Let ϕ m (t) and ϕ m i (t) be the intermediate data of a single dependency component m generated at multiple datacenters and the single home datacenter dc i respectively and, |ϕ m (t)| and |ϕ m i (t)| be their respective sizes.…”