High-Responsive Scheduling with MapReduce Performance Prediction on Hadoop YARN

Liu, Yang; Zeng, Yukun; Piao, Xuefeng

doi:10.1109/rtcsa.2016.51

Cited by 8 publications

(5 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…But the proposed scheduling did not consider node localization in allocation of reduce tasks to node. Liu et al [15] proposed a Fair Sojourn Protocol in YARN scheduler to improve the responsiveness and ensure fairness in Hadoop clusters. It is a size based scheduler where job size is predicted and based on job size, resources are allocated to it.…”

Section: B Task Managementmentioning

confidence: 99%

Integrated Data, Task and Resource Management to Speed Up Processing Small Files in Hadoop Cluster

2024

IJIES

View full text Add to dashboard Cite

Huge demand on business intelligence applications over large volume of enterprise data has resulted in rapid adoption of High performance data analytics. Hadoop based high performance computing environment are optimized for large files. Data centric execution with localization of computing proximal to data provided higher performance for large files. But for small files, the performance reduced and overhead increased due to the way Hadoop handles files. Resource allocation and scheduling policies of Hadoop has to be improvised to handle small files. This work proposes an integrated data, task, and resource management technique to speed up processing of small files in Hadoop based high performance computing environment. As part of data management, the data placement is made dynamic to access frequency and inherent data semantics. As part of task management, tasks were grouped based on fine grained data semantics correlation they process. As part of resource management, data blocks or storage nodes are replicated to improve the access latency with minimal cost overheads. With this integrated management, the proposed solution is able to increase the speedup of handling small files by at least 8.7% compared to most recent works on individual management of data, task and resources.

show abstract

Section: B Task Managementmentioning

confidence: 99%

Integrated Data, Task and Resource Management to Speed Up Processing Small Files in Hadoop Cluster

2024

IJIES

View full text Add to dashboard Cite

show abstract

“…YARN (Liu et al, 2016;Yao et al, 2021) is a resource management component proposed by Hadoop version 2.0. The Hadoop YARN system consists of multiple work nodes and resources, which are managed by a centralized Resource-Manager and multiple distributed NodeManagers.…”

Section: Construction Of a Hadoop Distributed Clustermentioning

confidence: 99%

A distributed data processing scheme based on Hadoop for synchrotron radiation experiments

Zhang,

Dai,

Sun

et al. 2024

J Synchrotron Radiat

View full text Add to dashboard Cite

With the development of synchrotron radiation sources and high-frame-rate detectors, the amount of experimental data collected at synchrotron radiation beamlines has increased exponentially. As a result, data processing for synchrotron radiation experiments has entered the era of big data. It is becoming increasingly important for beamlines to have the capability to process large-scale data in parallel to keep up with the rapid growth of data. Currently, there is no set of data processing solutions based on the big data technology framework for beamlines. Apache Hadoop is a widely used distributed system architecture for solving the problem of massive data storage and computation. This paper presents a set of distributed data processing schemes for beamlines with experimental data using Hadoop. The Hadoop Distributed File System is utilized as the distributed file storage system, and Hadoop YARN serves as the resource scheduler for the distributed computing cluster. A distributed data processing pipeline that can carry out massively parallel computation is designed and developed using Hadoop Spark. The entire data processing platform adopts a distributed microservice architecture, which makes the system easy to expand, reduces module coupling and improves reliability.

show abstract

“…To address the limitations of the keyword-matching approach, researchers have attempted to utilize natural language processing techniques for performance issue analysis. Liu et al [80] used topic modeling to identify performance-related issues in a Hadoop issue tracking system. Zaman et al [4] proposed a statistical text-mining approach that identifies performance-related issues based on specific patterns and keywords.…”

Section: Performance Issue Analysismentioning

confidence: 99%

A Platform-Agnostic Framework for Automatically Identifying Performance Issue Reports With Heuristic Linguistic Patterns

Zhao,

Xiao,

Wong

2024

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Software performance is critical for system efficiency, with performance issues potentially resulting in budget overruns, project delays, and market losses. Such problems are reported to developers through issue tracking systems, which are often under-tagged, as the manual tagging process is voluntary and time-consuming. Existing automated performance issue tagging techniques, such as keyword matching and machine/deep learning models, struggle due to imbalanced datasets and a high degree of variance. This paper presents a novel hybrid classification approach, combining Heuristic Linguistic Patterns (HLPs) with machine/deep learning models to enable practitioners to automatically identify performance-related issues. The proposed approach works across three progressive levels: HLP tagging, sentence tagging, and issue tagging, with a focus on linguistic analysis of issue descriptions. The authors evaluate the approach on three different datasets collected from different projects and issue-tracking platforms to prove that the proposed framework is accurate, project-and platform-agnostic, and robust to imbalanced datasets. Furthermore, this study also examined how the two unique techniques of the framework, including the fuzzy HLP matching and the Issue HLP Matrix, contribute to the accuracy. Finally, the study explored the effectiveness and impact of two off-the-shelf feature selection techniques, Boruta and RFE, with the proposed framework. The results showed that the proposed framework has great potential for practitioners to accurately (with up to 100% precision, 66% recall, and 79% F1-score) identify performance issues, with robustness to imbalanced data and good transferability to new projects and issue tracking platforms.

show abstract

High-Responsive Scheduling with MapReduce Performance Prediction on Hadoop YARN

Cited by 8 publications

References 14 publications

Integrated Data, Task and Resource Management to Speed Up Processing Small Files in Hadoop Cluster

Integrated Data, Task and Resource Management to Speed Up Processing Small Files in Hadoop Cluster

A distributed data processing scheme based on Hadoop for synchrotron radiation experiments

A Platform-Agnostic Framework for Automatically Identifying Performance Issue Reports With Heuristic Linguistic Patterns

Contact Info

Product

Resources

About