Jointly optimizing task granularity and concurrency for in-memory mapreduce frameworks

Bae, Jonghyun; Jang, Hakbeom; Jin, Wenjing; Heo, Jun; Jang, Jae-Young; Hwang, Joo-Young; Cho, Sangyeun; Lee, Jae Woo

doi:10.1109/bigdata.2017.8257921

Cited by 7 publications

(5 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Meanwhile, Spark continues to generate small heap objects, which occasionally get promoted to the oldgeneration heap, and requests extra space there. This triggers frequent major GCs [5].…”

Section: Background and Overviewmentioning

confidence: 99%

“…Comparison with WASP scheduler. We compare espill with WASP, a state-of-the-art Spark task scheduler [5]. WASP jointly optimizes both task granularity and parallelism based on workload characteristics.…”

Section: Performance Analysismentioning

confidence: 99%

“…Previous proposals address this challenge by i) adjusting the working set size by tuning task granularity and parallelism [5] or ii) moving large objects to outside the heap (i.e., JVM heap) [6]. In addition, the conventional in-memory processing systems provide spill-mechanism that serializes the partially created data and writes it to the local disk.…”

Section: Introductionmentioning

confidence: 99%

“…WASP scheduler[5] vs. e-spill Speedups across varying ratio of the time spent on spill to GC. All numbers are normalized to the baseline.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Eager Memory Management for In-Memory Data Analytics

Jang

Bae

Ham

et al. 2019

IEICE Trans. Inf. & Syst.

Self Cite

View full text Add to dashboard Cite

This paper introduces e-spill, an eager spill mechanism, which dynamically finds the optimal spill-threshold by monitoring the GC time at runtime and thereby prevent expensive GC overhead. Our e-spill adopts a slow-start model to gradually increase the spill-threshold until it reaches the optimal point without substantial GCs. We prototype e-spill as an extension to Spark and evaluate it using six workloads on three different parallel platforms. Our evaluations show that e-spill improves performance by up to 3.80× and saves the cost of cluster operation on Amazon EC2 cloud by up to 51% over the baseline system following Spark Tuning Guidelines.

show abstract

“…Meanwhile, Spark continues to generate small heap objects, which occasionally get promoted to the oldgeneration heap, and requests extra space there. This triggers frequent major GCs [5].…”

Section: Background and Overviewmentioning

confidence: 99%

Section: Performance Analysismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…WASP scheduler[5] vs. e-spill Speedups across varying ratio of the time spent on spill to GC. All numbers are normalized to the baseline.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Eager Memory Management for In-Memory Data Analytics

Jang

Bae

Ham

et al. 2019

IEICE Trans. Inf. & Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Spark provides rich operators and uses them to organize computational logic, but research on Spark operators is still relatively rare [6]. studied the input-output ratio of different operators, to estimate the size of intermediate data in computing process [7].…”

mentioning

confidence: 99%

A DAG Refactor Based Automatic Execution Optimization Mechanism for Spark

Zhao

Rong

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

In today's big data era, traditional disk-based MapReduce big data framework encountered bottlenecks due to its lower memory utilization and inefficient orchestration of complex tasks. With the advantage of fully use memory resources, Spark provides a lot of data manipulate operators and use DAG to express the dependences. Spark split entire job to multistage according to DAG and schedule them in a distributed execution environment, which better adapted to the new characteristic of big data processing. However, Spark didn't consider the resource requirement of different operators and schedule them indiscriminately, which could cause load imbalances on different nodes in the cluster and cause some node become bottlenecks due to its extraordinary resource consumption. In the past, solve this problem need developers to have a lot of experience of Spark and write code sophisticated. In this paper, we proposed a DAG refactor based automatic execution optimization mechanism for Spark. The experimental results show that the DAG refactor mechanism can greatly improve Spark performance by up to 8.8X without misinterpretation of original program semantics.

show abstract

AISFIP: Artificial Intelligence Smart Financial Information Platform with Concurrency Computation

Yang¹,

Yang

2022

2022 International Conference on Electronics and Renewable Systems (ICEARS)

View full text Add to dashboard Cite

Jointly optimizing task granularity and concurrency for in-memory mapreduce frameworks

Cited by 7 publications

References 19 publications

Eager Memory Management for In-Memory Data Analytics

Eager Memory Management for In-Memory Data Analytics

A DAG Refactor Based Automatic Execution Optimization Mechanism for Spark

AISFIP: Artificial Intelligence Smart Financial Information Platform with Concurrency Computation

Contact Info

Product

Resources

About