Learning automata-based algorithms for MapReduce data skewness handling

Irandoost, Mohammad Amin; Rahmani, Amir Masoud; Setayeshi, Saeed

doi:10.1007/s11227-019-02855-0

Cited by 6 publications

(2 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In MapReduce, reducer side data skew occurs due to unbalanced allocation of intermediate map-output to reducers. Therefore, [18] proposed an adaptive Learning Automata Hash Partitioning (LAHP) algorithm to address the data skew problem. The LAHP is based on learning automata game for conventional allocation of intermediate key-value pairs to designated reducers.…”

Section: Related Workmentioning

confidence: 99%

Straggler Mitigation in Hadoop MapReduce Framework: A Review

Ajibade¹,

Bakar²,

Aliyu³

et al. 2022

IJACSA

View full text Add to dashboard Cite

Processing huge and complex data to obtain useful information is challenging, even though several big data processing frameworks have been proposed and further enhanced. One of the prominent big data processing frameworks is MapReduce. The main concept of MapReduce framework relies on distributed and parallel processing. However, MapReduce framework is facing serious performance degradations due to the slow execution of certain tasks type called stragglers. Failing to handle stragglers causes delay and affects the overall job execution time. Meanwhile, several straggler reduction techniques have been proposed to improve the MapReduce performance. This study provides a comprehensive and qualitative review of the different existing straggler mitigation solutions. In addition, a taxonomy of the available straggler mitigation solutions is presented. Critical research issues and future research directions are identified and discussed to guide researchers and scholars.

show abstract

Section: Related Workmentioning

confidence: 99%

Straggler Mitigation in Hadoop MapReduce Framework: A Review

Ajibade¹,

Bakar²,

Aliyu³

et al. 2022

IJACSA

View full text Add to dashboard Cite

show abstract

“…However, this incurs an overhead associated with repartitioning the data and concatenating the output. In addition to the above solutions, Irandoost et al [31] propose the traffic cost-aware partitioner (TCAP) to handle reducerside data skewness in MapReduce. This approach attempts to balance the cost of network traffic during shuffling while balancing the reducer load; nevertheless, this approach only targets data skewness in the reduce stage.…”

Section: Related Workmentioning

confidence: 99%

Run-Time Dynamic Resource Adjustment for Mitigating Skew in MapReduce

Liu¹,

Zhang²,

Liu³

et al. 2021

Computer Modeling in Engineering &Amp; Sciences

View full text Add to dashboard Cite

MapReduce is a widely used programming model for large-scale data processing. However, it still suffers from the skew problem, which refers to the case in which load is imbalanced among tasks. This problem can cause a small number of tasks to consume much more time than other tasks, thereby prolonging the total job completion time. Existing solutions to this problem commonly predict the loads of tasks and then rebalance the load among them. However, solutions of this kind often incur high performance overhead due to the load prediction and rebalancing. Moreover, existing solutions target the partitioning skew for reduce tasks, but cannot mitigate the computational skew for map tasks. Accordingly, in this paper, we present DynamicAdjust, a run-time dynamic resource adjustment technique for mitigating skew. Rather than rebalancing the load among tasks, DynamicAdjust monitors the runtime execution of tasks and dynamically increases resources for those tasks that require more computation. In so doing, DynamicAdjust can not only eliminate the overhead incurred by load prediction and rebalancing, but also culls both the partitioning skew and the computational skew. Experiments are conducted based on a 21-node real cluster using real-world datasets. The results show that DynamicAdjust can mitigate the negative impact of the skew and shorten the job completion time by up to 40.85%.

show abstract