MapReduce Data Skewness Handling: A Systematic Literature Review

Irandoost, Mohammad Amin; Rahmani, Amir Masoud; Setayeshi, Saeed

doi:10.1007/s10766-019-00627-0

Cited by 11 publications

(7 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, it cannot guarantee the high performance of parallel computing due to the data skewness issue. 41 RQ4: What are the open perspectives, challenges, and future research directions of data mining in fog computing?…”

Section: Discussion and Comparisonmentioning

confidence: 99%

“…The scalability and performance of the MapReduce‐based frameworks indicated the importance of this method for pattern mining in the fog computing environments. However, it cannot guarantee the high performance of parallel computing due to the data skewness issue 41 …”

Section: Discussion and Comparisonmentioning

confidence: 99%

“…It is highly recommended to consider this point since addressing data skewness in the MapReduce framework can decrease make‐span and increase performance. Data skewness could be an open issue and the main challenge in this domain 41 …”

Section: Discussion and Comparisonmentioning

confidence: 99%

See 2 more Smart Citations

Frequent pattern mining algorithms in fog computing environments: A systematic review

Tehrani

Sharifi

Rahmani

2022

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

Recent advances in technology have resulted in generating or collecting massive volumes of data from rich data resources such as sensors and mobile devices in Internet of Things (IoT). Using data mining techniques can help overcome the mining problem in Fog computing environments which include millions of IoT devices.In addition, it can optimize response times, recourse consumption, and scalability in IoT applications. Frequent pattern mining, as one of the fundamental data mining tasks, is used for finding hidden patterns in such large datasets. The traditional data mining algorithms have many challenges such as scalability and resource consumption. This systematic review aimed to investigate the data mining algorithms, which focus on handling massive datasets, and present a technical taxonomy including the transaction-centric, item-centric, distributed, and parallel topics. The transaction-centric and MapReduce-based approaches were mostly utilized by 37% and 38%, respectively. Additionally, item-centric, distributed, and parallel algorithms were employed 12% and 13%, respectively. The response time as a Quality of Service (QoS) factor had the highest percentage in the estimations of data mining algorithms (55%), followed by scalability (25%), and cost (20%). To the best of our knowledge, no study has focused on fog-computing frequent pattern mining algorithms as one of the most important data mining tasks. This article aims to present a systematic review of the frequent pattern mining algorithms in fog computing and discuss the issues, challenges, and research perspectives for helping academia and industry leverage the power of data mining algorithms in fog computing.

show abstract

Section: Discussion and Comparisonmentioning

confidence: 99%

Section: Discussion and Comparisonmentioning

confidence: 99%

See 1 more Smart Citation

Frequent pattern mining algorithms in fog computing environments: A systematic review

Tehrani

Sharifi

Rahmani

2022

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

show abstract

“… What is the role of RA in cloud computing and its importance for cloud computing researchers? What modules does a resource management system (RMS) consist of? Based on what parameters can an RA mechanism be categorised? How many recent research studies have been made in the field of RA? What tools have researchers used to evaluate the results of RA mechanisms? What challenges and open issues remain in the field of RA and what issues are currently covered? This research started in 2016. The primary methodology and strategy of this paper are based on our previous review papers [4–16 ]. For a systematic presentation of this paper, the mechanisms have been proposed based on their published years and their databases.…”

Section: Methodsmentioning

confidence: 99%

Resource allocation mechanisms in cloud computing: a systematic literature review

et al. 2020

Self Cite

View full text Add to dashboard Cite

Cloud computing offers a vast number of processing opportunities and heterogeneous resources and meets the requirements of numerous applications at various levels. Thus, the allocation and management of resources are vital in cloud computing. Resource allocation is a technique in which the available resources such as central processing unit, random‐access memory, storage, and network bandwidth in cloud data centres are divided among users in a way that facilitates resource utilisation, provider profit, and user satisfaction. Integration and interaction with other modules of the resource management system, security, privacy, fairness, non‐fragmentation of resources, resource utilisation, provider profit, user satisfaction, reducing energy consumption, load balancing, flexibility, scalability, availability, improvement the number and time of virtual machine migrations, and the number of overloaded resources are considered as challenges for the resource allocation mechanism. A systematic resource allocation survey with innovations in resource management system architecture, categorising mechanisms, addressing the challenges, and issues is presented. In addition to introducing the existing resource allocation mechanisms, other similar survey papers have been reviewed. Finally, there are some suggested topics for future work.

show abstract

“…Data skew is important for the performance of processing big data. Mohammad et al [1] surveyed data skew from 2010 to 2017. They divided the data skew problem into four types: map stage, reduce stage, map & reduce stage, and shuffle stage, according to data skew.…”

Section: Related Workmentioning

confidence: 99%

Scheduling Spark Tasks With Data Skew and Deadline Constraints

2021

IEEE Access

View full text Add to dashboard Cite

Data skew has an essential impact on the performance of big data processing. Spark task scheduling with data skew and deadline constraints is considered to minimize the total rental cost in this paper. A modified scheduling architecture is developed in terms of the unique characteristics of the considered problem. A mathematical model is constructed, and a Spark task scheduling algorithm is proposed considering both the data skew and deadline constraints. The algorithm consists of three components: stage sequencing, task scheduling, and scheduling adjustment. Strategies for each of the components are presented. The parameters and components of the proposed algorithm are calibrated over many random instances. The calibrated algorithm is compared to two existing algorithms for similar problems over classical scientific workflow applications. Experimental results show that the proposed algorithm outperforms the compared algorithms statistically.

show abstract

MapReduce Data Skewness Handling: A Systematic Literature Review

Cited by 11 publications

References 43 publications

Frequent pattern mining algorithms in fog computing environments: A systematic review

Frequent pattern mining algorithms in fog computing environments: A systematic review

Resource allocation mechanisms in cloud computing: a systematic literature review

Scheduling Spark Tasks With Data Skew and Deadline Constraints

Contact Info

Product

Resources

About