Big data framework interference in restricted private cloud settings

Dimopoulos, Stratos; Krintz, Chandra; Wolski, Rich

doi:10.1109/bigdata.2016.7840620

Cited by 8 publications

(2 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We want to use the limited resources of the cloudlet only for the workloads that absolutely need them in order to satisfy their latency requirements and send the workloads with more relaxed constraints to run on the virtually unlimited resources of a remote cloud or to less Figure 2: Justice in a multi-cloudlet environment loaded neighboring cloudlets. We suggest a distributed scheduling scheme to coordinate resource allocation across cloudlets, inspired by two-level resource allocators for big data analytics like Apache Mesos [10] but specialized to overcome framework interference and fairness violation issues as these systems were not designed with resource-scarcity in mind [4]. Our work is also inspired by application specific cloudlets and multi-cloud environments [16], as this can be an extra incentive for collaboration between cloudlets.…”

Section: Background and Related Workmentioning

confidence: 99%

Towards distributed, fair, deadline-driven resource allocation for cloudlets

Dimopoulos

Krintz

Wolski

2019

Proceedings of the 4th Workshop on Middleware for Edge Clouds &Amp; Cloudlets - MECC '19

Self Cite

View full text Add to dashboard Cite

In this paper we present our vision for a two-level, distributed resource allocator that preserves fairness and satisfies deadlines of low latency workloads in a multi-cloudlet environment with offloading support. We analyze the opportunities and challenges that offloading and the multi-cloud environment impose and we suggest the changes required to a fair-preserving and deadline-driven resource allocator originally designed for resource-constrained environments. CCS CONCEPTS • Computer systems organization → Cloud computing.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Towards distributed, fair, deadline-driven resource allocation for cloudlets

Dimopoulos

Krintz

Wolski

2019

Proceedings of the 4th Workshop on Middleware for Edge Clouds &Amp; Cloudlets - MECC '19

Self Cite

View full text Add to dashboard Cite

show abstract

“…Several studies have been conducted to evaluate the performance of distributed data processing frameworks in private and public clouds. Mavridis et al [9] and Dimopoulos et al [10] have compared the performance of Hadoop and Spark in private clouds. In [11], the authors have compared Hadoop with Spark deployed in public clouds.…”

Section: Introductionmentioning

confidence: 99%

Evaluation of Distributed Data Processing Frameworks in Hybrid Clouds

Ullah¹,

Dhingra²,

Xia³

et al. 2022

Preprint

View full text Add to dashboard Cite

Distributed data processing frameworks (e.g., Hadoop, Spark, and Flink) are widely used to distribute data among computing nodes of a cloud. Recently, there have been increasing efforts aimed at evaluating the performance of distributed data processing frameworks hosted in private and public clouds. However, there is a paucity of research on evaluating the performance of these frameworks hosted in a hybrid cloud, which is an emerging cloud model that integrates private and public clouds to use the best of both worlds. Therefore, in this paper, we evaluate the performance of Hadoop, Spark, and Flink in a hybrid cloud in terms of execution time, resource utilization, horizontal scalability, vertical scalability, and cost. For this study, our hybrid cloud consists of OpenStack (private cloud) and MS Azure (public cloud). We use both batch and iterative workloads for the evaluation. Our results show that in a hybrid cloud (i) the execution time increases as more nodes are borrowed by the private cloud from the public cloud, (ii) Flink outperforms Spark, which in turn outperforms Hadoop in terms of execution time, (iii) Hadoop transfers the largest amount of data among the nodes during the workload execution while Spark transfers the least amount of data, (iv) all three frameworks horizontally scale better as compared to vertical scaling, and (v) Spark is found to be least expensive in terms of $ cost for data processing while Hadoop is found the most expensive.

show abstract