MicroRank: End-to-End Latency Issue Localization with Extended Spectrum Analysis in Microservice Environments

Yu, Guangba; Chen, Pengfei; Chen, Hongyang; Guan, Zijie; Huang, Zicheng; Jing, Linxiao; Weng, Tianjun; Sun, Xinmeng; Li, Xiaoyun

doi:10.1145/3442381.3449905

Cited by 61 publications

(12 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently there have been works on rootcause analysis or anomaly localization in Micro-service based systems, e.g. [8], [9] where the goal is to find the failing microservice among the services.…”

Section: Introductionmentioning

confidence: 99%

Pinpointing Anomaly Events in Logs from Stability Testing -- N-Grams vs. Deep-Learning

Mäntylä¹,

Varela²,

Hashemi³

2022

Preprint

View full text Add to dashboard Cite

As stability testing execution logs can be very long, software engineers need help in locating anomalous events. We develop and evaluate two models for scoring individual logevents for anomalousness, namely an N-Gram model and a Deep Learning model with LSTM (Long short-term memory). Both are trained on normal log sequences only. We evaluate the models with long log sequences of Android stability testing in our company case and with short log sequences from HDFS (Hadoop Distributed File System) public dataset. We evaluate next event prediction accuracy and computational efficiency. The LSTM model is more accurate in stability testing logs (0.848 vs 0.865), whereas in HDFS logs the N-Gram is slightly more accurate (0.904 vs 0.900). The N-Gram model has far superior computational efficiency compared to the Deep model (4 to 13 seconds vs 16 minutes to nearly 4 hours), making it the preferred choice for our case company. Scoring individual log events for anomalousness seems like a good aid for root cause analysis of failing test cases, and our case company plans to add it to its online services. Despite the recent surge in using deep learning in software system anomaly detection, we found limited benefits in doing so. However, future work should consider whether our finding holds with different LSTM-model hyper-parameters, other datasets, and with other deep-learning approaches that promise better accuracy and computational efficiency than LSTM based models.

show abstract

Section: Introductionmentioning

confidence: 99%

Pinpointing Anomaly Events in Logs from Stability Testing -- N-Grams vs. Deep-Learning

Mäntylä¹,

Varela²,

Hashemi³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…To measure the performance of the estimated graph and truth graph, we can compute the precision and recall of the adjacent confusion matrix and the Structural Hamming Distance (SHD), which is computed as wrongly directed edges. To measure the performance of the root cause localization, we can calculate the recall with the top-k results, which is widely used in existing works (Meng et al, 2020;Li et al, 2022;Yu et al, 2021). These related evaluation metrics are implemented in pyrca.utils.evalution.…”

Section: Output Layermentioning

confidence: 99%

AI Empowered Net-RCA for 6G

Qiu¹,

Yang²,

Wang³

et al. 2023

IEEE Network

View full text Add to dashboard Cite

We introduce PyRCA, an open-source Python machine learning library of Root Cause Analysis (RCA) for Artificial Intelligence for IT Operations (AIOps). It provides a holistic framework to uncover the complicated metric causal dependencies and automatically locate root causes of incidents. It offers a unified interface for multiple commonly used RCA models, encompassing both graph construction and scoring tasks. This library aims to provide IT operations staff, data scientists, and researchers a one-step solution to rapid model development, model evaluation and deployment to online applications. In particular, our library includes various causal discovery methods to support causal graph construction, and multiple types of root cause scoring methods inspired by Bayesian analysis, graph analysis and causal analysis, etc. Our GUI dashboard offers practitioners an intuitive pointand-click interface, empowering them to easily inject expert knowledge through human interaction. With the ability to visualize causal graphs and the root cause of incidents, practitioners can quickly gain insights and improve their workflow efficiency. This technical report introduces PyRCA's architecture and major functionalities, while also presenting benchmark performance numbers in comparison to various baseline models. Additionally, we demonstrate PyRCA's capabilities through several example use cases.

show abstract

“…Trace Anomaly Score: The Spectrum-based fault localization (SBFL) technique 25,26 has been widely used in software testing, which is mainly based on the coverage information of current program elements to assess the degree of program anomaly. Inspired by this, we calculate the anomaly score by using the SBFL technique, which is also mentioned in previous approaches 9,11,27,28 .…”

Section: Notation Definitionsmentioning

confidence: 99%

“…Ye et al 9 proposed T-Rank that used a spectrum analysis algorithm to locate root causes of abnormal services. Yu et al 27 proposed MicroRank. It firstly differentiates traces, then uses the PageRank algorithm to analyze the importance of different traces, and finally uses spectrum analysis to locate root cause of abnormal services.…”

Section: Related Workmentioning

confidence: 99%

Graph-Based Root Cause Localization in Microservice Systems with Protection Mechanisms

Zhang

Tian

Yang

et al. 2022

Preprint

View full text Add to dashboard Cite

Nowadays, the protection mechanisms are introduced into microservice systems to ensure the stable operation of services. However, existing approaches ignore the impact of protection mechanisms on the root cause localization of abnormal services. Specifically, the circuit breaking and rate limiting mechanisms can refuse service requests and thus change the way of anomaly propagation. Moreover, different service request frequencies and response time make service dependencies change dynamically, resulting in different probabilities of anomaly propagation among services. In this paper, we propose a novel framework named MicroGBPM to locate the root cause of abnormal services, which considers the impact of the protection mechanisms. We model anomaly propagation among services as a dynamically constructed service attributed graph with metrics and traces when a failure occurs. To eliminate the impact of the protection mechanisms, we design a two-stage dynamic calibration strategy to adjust the probability of anomaly propagation among services. Then we propose a random walking approach to calculate the root cause results by using the PageRank algorithm. The experimental results show that MicroGBPM improves the accuracy of root cause localization compared to other approaches in microservice systems with protection mechanisms.

show abstract

MicroRank: End-to-End Latency Issue Localization with Extended Spectrum Analysis in Microservice Environments

Cited by 61 publications

References 38 publications

Pinpointing Anomaly Events in Logs from Stability Testing -- N-Grams vs. Deep-Learning

Pinpointing Anomaly Events in Logs from Stability Testing -- N-Grams vs. Deep-Learning

AI Empowered Net-RCA for 6G

Graph-Based Root Cause Localization in Microservice Systems with Protection Mechanisms

Contact Info

Product

Resources

About