TaskTracker Aware Scheduler with Resource Availability Control for Hadoop MapReduce

Manjaly, Jisha S.; Subbulakshmi, T.

doi:10.1504/ijaip.2018.10017428

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is only one Job Tracker in the entire cluster, and it is easy to have a single point of failure when the resource consumption is so severe [24]. Hadoop2.x version designed a new resource scheduler, Yarn, also known as MRv2 [25].…”

Section: Figure 1 Structure Of Hdfsmentioning

confidence: 99%

Research on the Deployment Strategy of Big Data Visualization Platform by the Internet of Things Technology

Zhang

2023

EAI Endorsed Scal Inf Syst

View full text Add to dashboard Cite

INTRODUCTION: To improve the big data visualization platform's performance and task scheduling capability, a big data visualization platform is constructed based on Field Programmable Gate Array (FPGA) chip application power equipment.OBJECTIVES: This study proposes to combine a genetic algorithm and an ant colony scheduling (ACOS) algorithm to design a big data visualization platform deployment strategy based on an improved ACOS algorithm.METHODS: Firstly, big data technology is analyzed. Then, the basic theory of the ant colony algorithm is studied. According to the basic theory of ACOS and genetic algorithm, an improved ACOS algorithm model is constructed. The improved ACOS algorithm scheduler is compared with the other three schedulers. Under the same environment, the completion time of scheduling the same job and different task amounts are analyzed. The Central Processing Unit (CPU) utilization is analyzed when different schedulers have entirely different workloads. RESULTS: The results show that the constructed big data visualization platform based on the improved ACOS algorithm model has higher task scheduling efficiency than other schedulers and can greatly shorten the data processing time. The experimental results show that under the homogeneous cluster, the completion time of the improved ACOS algorithm generally lags the capacity scheduler and the fair scheduler. Under the heterogeneous cluster, the improved ACOS algorithm scheduler can reasonably allocate tasks to nodes with different performances, reducing the task completion time. When the number of completed tasks increases from 50 to 200, the time increases by 45s, and the completion time is shorter than other schedulers. The CPU utilization of different task volumes is the highest, and the utilization rate increases from 81% to 95%. CONCLUSION: The improved ACOS algorithm scheduler has the shortest data processing time and the highest efficiency. This work provides a specific reference value for optimizing the big data visualization platform's deployment strategy and improving the platform's performance.

show abstract

Section: Figure 1 Structure Of Hdfsmentioning

confidence: 99%

Research on the Deployment Strategy of Big Data Visualization Platform by the Internet of Things Technology

Zhang

2023

EAI Endorsed Scal Inf Syst

View full text Add to dashboard Cite

show abstract

TaskTracker Aware Scheduler with Resource Availability Control for Hadoop MapReduce

Cited by 1 publication

References 0 publications

Research on the Deployment Strategy of Big Data Visualization Platform by the Internet of Things Technology

Research on the Deployment Strategy of Big Data Visualization Platform by the Internet of Things Technology

Contact Info

Product

Resources

About