Improving Cache Management with Redundant RDDs Eviction in Spark

Zhao, Yao; Dong, Jian; Liu, Hongwei; Wu, Jin; Liu, Yanxin

doi:10.32604/cmc.2021.016462

Cited by 1 publication

(1 citation statement)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[18] proposed an AWRP weight replacement algorithm, which obtains the weight of each data block by calculating the occurrence frequency of data blocks, the last time of use and the total number of accesses. Spark cache replacement strategy optimization based on DAG is another research direction [19][20][21]. Yu et al proposed an LRC cache replacement scheme in [22], which obtained a DAG graph according to the RDD dependencies, and then used the number of dependencies of each RDD as the selection basis for the cache replacement strategy.…”

Section: Introductionmentioning

confidence: 99%

A Memory-Aware Spark Cache Replacement Strategy

Zhang¹,

Zhang²,

Zhang³

et al. 2022

Journal of Internet Technology

View full text Add to dashboard Cite

<p>Spark is currently the most widely used distributed computing framework, and its key data abstraction concept, Resilient Distributed Dataset (RDD), brings significant performance improvements in big data computing. In application scenarios, Spark jobs often need to replace RDDs due to insufficient memory. Spark uses the Least Recently Used (LRU) algorithm by default as the cache replacement strategy. This algorithm only considers the most recent use time of RDDs as the replacement basis. This characteristic may cause the RDDs that need to be reused to be evicted when performing cache replacement, resulting in a decrease in Spark performance. In response to the above problems, this paper proposes a memory-aware Spark cache replacement strategy, which comprehensively considers the cluster memory usage, RDD size, RDD dependencies, usage times and other information when performing cache replacement and selects the RDDs to be evicted. Furthermore, this paper designs extensive corresponding experiments to test and analyze the performance of the memory-aware Spark cache replacement strategy. The experimental data show that the proposed strategy can improve the performance by up to 13% compared with the LRU algorithm in different scenarios.</p> <p> </p>

show abstract