Pado

Yang, Yooseong; Kim, Geon Woo; Song, Won Wook; Lee, Yunseong; Chung, Andrew; Qian, Zhuzhong; Cho, Brian; Chun, Byung-Gon

doi:10.1145/3064176.3064181

Cited by 37 publications

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Probability distribution based resource management for multitenant cloud clusters

Zhou

Feng

Wang

2021

Concurrency and Computation

View full text Add to dashboard Cite

Accurate resource allocation and colocating jobs are effective ways to increase resource utilization and reduce costs in modern datacenters. The main challenges are fluctuations in resource consumption and interference among colocated jobs. Therefore, we propose a resource management schema based on the probability distributions of resource consumption and completion time for multitenant cloud clusters. First, we found that the characteristics of the task can be well described by the probability distribution of resource consumption and completion time, and the probability distribution function can be obtained by Gaussian fitting. Second, we propose a probability distribution based resource allocation (PDRA) strategy for batch jobs. Third, we design a tail latency aware allocation (TLAA) strategy to use transient resources efficiently while ensuring tail latency requirements. Finally, we design a cost‐effective resource revocation (CERR) strategy to revoke transient resources with minimal eviction costs. Experimental results demonstrate the efficiency of our resource management. Our resource allocation strategies (PDRA and TLAA) can effectively improve resource utilization and reduce job completion time. CERR can reduce the impact of resource revocation on batch jobs and perform better than existing resource revocation strategies.

show abstract

Probability distribution based resource management for multitenant cloud clusters

Zhou

Feng

Wang

2021

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

iStore: Towards the Optimization of Federation File Systems

Khan¹,

Attique²,

Kim³

2019

IEEE Access

View full text Add to dashboard Cite

With the growing volumes of data, many organizations are deploying geo-distributed edge servers and building federations atop of edge servers to improve data sharing, effective collaborations, and analytics. Multiple federation file systems are designed to satisfy such needs, but due to application-specific architectures, these federations neglect some important features that can improve the overall federation performance. In this paper, we address the important challenges of federated file systems, in particular, global namespace, optimal data placement and analysis, efficient data migration across edge servers, and metadata optimizations. To further investigate these challenges, we prototyped the federation file system iStore to emulate the federation and showed the significance of the afore-mentioned key challenges in the federation. The iStore provides unified global namespace atop of geo-distributed edge servers with a generic job and resource-aware data storage and placement algorithm (JRAP), which minimizes the job execution time by considering resources at each edge server. Furthermore, to enable effective data migration, we employed direct channel file layout-aware data transfer and designed a batch-based metadata scheme for federations to reduce the metadata contention with increasing clients. We evaluated the efficacy of various big data applications from data generation to storage and analysis using the iStore on real testbed and simulation. INDEX TERMS Big data storage and HPC, geo-distributed edge computing, cluster storage and analysis.

show abstract