BigRoots: An Effective Approach for Root-Cause Analysis of Stragglers in Big Data System

Zhou, Honggang; Li, Yun‐Chun; Yang, Hailong; Jia, Jie; Li, Wei

doi:10.1109/access.2018.2859826

Cited by 20 publications

(10 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The authors in [28] use offline log analysis methods to identify the root cause of outliers in a large-scale cluster consisting of thousands of nodes by tracking the resource utilization. Similarly, Zhou et al [29] use a simple but efficient rule based method to identify the root cause of stragglers. Along with these similar works, there are some researchers using statistical and machine learning methods for root-cause analysis.…”

Section: Related Workmentioning

confidence: 99%

AutoDiagn: An Automated Real-Time Diagnosis Framework for Big Data Systems

Demirbaga

Wen

Noor

et al. 2022

IEEE Trans. Comput.

View full text Add to dashboard Cite

Big data processing systems, such as Hadoop and Spark, usually work in large-scale, highly-concurrent, and multi-tenant environments that can easily cause hardware and software malfunctions or failures, thereby leading to performance degradation. Several systems and methods exist to detect big data processing systems' performance degradation, perform root-cause analysis, and even overcome the issues causing such degradation. However, these solutions focus on specific problems such as stragglers and inefficient resource utilization. There is a lack of a generic and extensible framework to support the real-time diagnosis of big data systems. In this paper, we propose, develop and validate AutoDiagn. This generic and flexible framework provides holistic monitoring of a big data system while detecting performance degradation and enabling root-cause analysis. We present an implementation and evaluation of AutoDiagn that interacts with a Hadoop cluster deployed on a public cloud and tested with real-world benchmark applications. Experimental results show that AutoDiagn can offer a high accuracy root-cause analysis framework, at the same time as offering a small resource footprint, high throughput and low latency.

show abstract

Section: Related Workmentioning

confidence: 99%

AutoDiagn: An Automated Real-Time Diagnosis Framework for Big Data Systems

Demirbaga

Wen

Noor

et al. 2022

IEEE Trans. Comput.

View full text Add to dashboard Cite

show abstract

“…Many reasons for such stragglers to occur including load imbalance, scheduling inefficiencies, data locality, communication overheads hardware heterogeneity [12,13]. There have also been efforts looking to address one or more of these concerns to mitigate the problem [14][15][16]. Sesipite all of these prior efforts are important and useful in overcoming this problem, we believe that a rigorous set of analytical tools is needed in order to better understand the consequences of stragglers on the performance slowdown in big data [17,18].…”

Section: Mapreduce Framework and Stragglersmentioning

confidence: 99%

Straggler handling approaches in mapreduce framework: a comparative study

Katrawi

Abdullah

Anbar

et al. 2021

IJECE

View full text Add to dashboard Cite

The proliferation of information technology produces a huge amount of data called big data that cannot be processed by traditional database systems. These Various types of data come from different sources. However, stragglers are a major bottleneck in big data processing, and hence the early detection and accurate identification of stragglers can have important impacts on the performance of big data processing. This work aims to assess five stragglers identification methods: Hadoop native scheduler, LATE Scheduler, Mantri, MonTool, and Dolly. The performance of these techniques was evaluated based on three benchmarked methods: Sort, Grep and WordCount. The results show that the LATE Scheduler performs the best and it would be efficient to obtain better results for stragglers identification.

show abstract

“…Such jobs and subsequent tasks are scheduled onto different machines in a parallelized manner to accelerate job completion and are often divided into phases creating a Direct Acyclic Graph (DAG) [83]. Application frameworks (such as MapReduce) attempt to sub-divide jobs so that tasks will approximately complete within the same timeframe for each phase [84]. This is achieved by providing a sub-set of data (known as shards) to each task, and allocating the appropriate resources to tasks (CPU, memory, etc).…”

Section: Straggler Definition and Impactmentioning

confidence: 99%

Tails in the cloud: a survey and taxonomy of straggler management within large-scale cloud data centres

2020

View full text Add to dashboard Cite

Cloud computing systems are splitting compute and data intensive jobs into smaller tasks to execute them in a parallel manner using clusters to improve execution time. However, such systems at increasing scale are exposed to stragglers, whereby abnormally slow running tasks executing within a job substantially affect job performance completion. Such stragglers are a direct threat towards attaining fast execution of data intensive jobs within cloud computing. Researchers have proposed an assortment of different mechanisms, frameworks, and management techniques to detect and mitigate stragglers both proactively and reactively. In this paper, we present a comprehensive review of straggler management techniques within large-scale cloud datacenters. We provide a detailed taxonomy of straggler causes, as well as proposed management and mitigation techniques based on straggler characteristics and properties. From this systematic review, we outline several outstanding challenges and potential directions of possible future work for straggler research.

show abstract

BigRoots: An Effective Approach for Root-Cause Analysis of Stragglers in Big Data System

Cited by 20 publications

References 16 publications

AutoDiagn: An Automated Real-Time Diagnosis Framework for Big Data Systems

AutoDiagn: An Automated Real-Time Diagnosis Framework for Big Data Systems

Straggler handling approaches in mapreduce framework: a comparative study

Tails in the cloud: a survey and taxonomy of straggler management within large-scale cloud data centres

Contact Info

Product

Resources

About