LADRA: Log-based abnormal task detection and root-cause analysis in big data processing with Spark

Lu, Siyang; Wei, Xiang; Rao, Bingbing; Tak, Byungchul; Wang, Long; Wang, Liqiang

doi:10.1016/j.future.2018.12.002

Cited by 29 publications

(12 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Along with these similar works, there are some researchers using statistical and machine learning methods for root-cause analysis. The authors of [30] introduce a Regression Neural Network (RNN) based algorithm to troubleshoot the causes of stragglers by processing Spark logs. More algorithms such as the associated tree and fuzzy data envelopment analysis [31] and Reinforcement Learning [32] are applied for finding the reasons of stragglers in Hadoop and Spark.…”

Section: Related Workmentioning

confidence: 99%

AutoDiagn: An Automated Real-Time Diagnosis Framework for Big Data Systems

Demirbaga

Wen

Noor

et al. 2022

IEEE Trans. Comput.

View full text Add to dashboard Cite

Big data processing systems, such as Hadoop and Spark, usually work in large-scale, highly-concurrent, and multi-tenant environments that can easily cause hardware and software malfunctions or failures, thereby leading to performance degradation. Several systems and methods exist to detect big data processing systems' performance degradation, perform root-cause analysis, and even overcome the issues causing such degradation. However, these solutions focus on specific problems such as stragglers and inefficient resource utilization. There is a lack of a generic and extensible framework to support the real-time diagnosis of big data systems. In this paper, we propose, develop and validate AutoDiagn. This generic and flexible framework provides holistic monitoring of a big data system while detecting performance degradation and enabling root-cause analysis. We present an implementation and evaluation of AutoDiagn that interacts with a Hadoop cluster deployed on a public cloud and tested with real-world benchmark applications. Experimental results show that AutoDiagn can offer a high accuracy root-cause analysis framework, at the same time as offering a small resource footprint, high throughput and low latency.

show abstract

Section: Related Workmentioning

confidence: 99%

AutoDiagn: An Automated Real-Time Diagnosis Framework for Big Data Systems

Demirbaga

Wen

Noor

et al. 2022

IEEE Trans. Comput.

View full text Add to dashboard Cite

show abstract

“…The anomalies are detected on a per task basis, by comparing the tasks to the average and standard deviation of task time in each stage. However, since the statistical approach does not provide good precision on classifying the root cause, the authors extend their work in a follow up paper [6]. In the extension, the authors develop a system that analyzes Spark application execution logs to identify resource bottlenecks, e.g., processor, memory, disk, and network, causing abnormally high execution times.…”

Section: Related Workmentioning

confidence: 99%

Diaspore: Diagnosing Performance Interference in Apache Spark

2021

View full text Add to dashboard Cite

Apache Spark is being increasingly used to execute big data applications on cluster computing platforms. To increase system utilization, cluster operators often configure their clusters such that multiple co-located applications can simultaneously share the resources of a cluster node. With resource sharing, applications can compete with each other for shared node resources thereby interfering with each other's performance. Many Spark applications take a long time to execute. Performance interference from other applications can thus cause a Spark application to fail or take even longer time to execute thereby wasting cluster resources and frustrating users. This motivates the need for an automated technique that can detect interference quickly and also diagnose the root cause of the interference to facilitate mitigation of the problem. Most existing approaches are not designed to offer quick interference detection and diagnosis. For example, they typically require extensive training data for every application of interest under various possible input data sizes and resource allocations. In this paper, we systematically investigate the design of a Machine Learning (ML) based technique that addresses this open problem. We implement a tool called Diaspore that integrates our findings. We evaluate the tool with a diverse set of 13 Spark applications executing on a real cluster. Experimental results show that Diaspore requires only small scale training data, i.e., executions under small input sizes and resource allocations. Furthermore, our results show that the tool can offer accurate predictions for applications not present in the training data. Consequently, Diaspore reduces the training time needed to offer predictions. Finally, the feature engineering underlying Diaspore ensures that the tool can detect and diagnose interference quickly in an online manner by sampling only a small fraction of a long running application's execution. This can allow cluster operators to mitigate interference in an agile manner.

show abstract

“…Monitoring data may not be always accessible from the user side since the monitoring tools are hard to install and tune. Hence, some studies focus on the offline strategy by analyzing logs instead of monitoring Lu et al [5]. Cluster managers, e.g., YARN in Vavilapalli et al [6], Isard et al [7], Verma et al [8], have different focuses.…”

Section: Introductionmentioning

confidence: 99%

Collaborative Learning Based Straggler Prevention in Large-Scale Distributed Computing Framework

Deshmukh

Rao

Shabaz

2021

Security and Communication Networks

View full text Add to dashboard Cite

Modern big data applications tend to prefer a cluster computing approach as they are linked to the distributed computing framework that serves users jobs as per demand. It performs rapid processing of tasks by subdividing them into tasks that execute in parallel. Because of the complex environment, hardware and software issues, tasks might run slowly leading to delayed job completion, and such phenomena are also known as stragglers. The performance improvement of distributed computing framework is a bottleneck by straggling nodes due to various factors like shared resources, heavy system load, or hardware issues leading to the prolonged job execution time. Many state-of-the-art approaches use independent models per node and workload. With increased nodes and workloads, the number of models would increase, and even with large numbers of nodes. Not every node would be able to capture the stragglers as there might not be sufficient training data available of straggler patterns, yielding suboptimal straggler prediction. To alleviate such problems, we propose a novel collaborative learning-based approach for straggler prediction, the alternate direction method of multipliers (ADMM), which is resource-efficient and learns how to efficiently deal with mitigating stragglers without moving data to a centralized location. The proposed framework shares information among the various models, allowing us to use larger training data and bring training time down by avoiding data transfer. We rigorously evaluate the proposed method on various datasets with high accuracy results.

show abstract

LADRA: Log-based abnormal task detection and root-cause analysis in big data processing with Spark

Cited by 29 publications

References 6 publications

AutoDiagn: An Automated Real-Time Diagnosis Framework for Big Data Systems

AutoDiagn: An Automated Real-Time Diagnosis Framework for Big Data Systems

Diaspore: Diagnosing Performance Interference in Apache Spark

Collaborative Learning Based Straggler Prevention in Large-Scale Distributed Computing Framework

Contact Info

Product

Resources

About