Practical Root Cause Localization for Microservice Systems via Trace Analysis

Li, Zeyan; Chen, Junjie; Jiao, Rui; Zhao, Ning; Wang, Zhijun; Zhang, Shuwei; Wu, Yicheng; Jiang, Long; Yan, Leiqin; Wang, Zikai; Chen, Zhekang; Zhang, Wenchi; Nie, Xin; Sui, Kaixin; Pei, Dan

doi:10.1109/iwqos52092.2021.9521340

Cited by 73 publications

(7 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [5] , An anomaly detection method ADS for container-based microservice performance metrics is proposed, which considers a set of key performance indicators for each containerized service and then uses a machine learning algorithm to train a classifier for detecting the indicator conditions. [6] Use a range of system status data such as per-call latency, HTTP status, etc. as metrics for tracing anomaly detection, and adaptively select useful metrics for each anomaly to improve monitoring accuracy.…”

Section: Related Workmentioning

confidence: 99%

Trace-based microservice anomaly detection through deep learning

Bai

Zhang

2023

Second International Conference on Electronic Information Engineering, Big Data, and Computer Technology (EIBDCT 2023)

View full text Add to dashboard Cite

Over the years, microservices have become increasingly popular and are being adopted more and more, microservice architecture is gradually replacing monolithic applications as the mainstream architecture. Anomaly detection for microservices is a hot topic of current research, which is important to ensure the security and performance of the application. This paper proposes a deep learning based anomaly detection method for microservices, which first serializes trace data, then trains a model combining AE and LSTM to detect anomalies, and also detects performance anomalies by executing time series. We evaluated our approach on a widely adopted microservice system and showed that our approach outperformed current trace-based approaches.

show abstract

Section: Related Workmentioning

confidence: 99%

Trace-based microservice anomaly detection through deep learning

Bai

Zhang

2023

Second International Conference on Electronic Information Engineering, Big Data, and Computer Technology (EIBDCT 2023)

View full text Add to dashboard Cite

show abstract

“…N, n the number of traces, the number of services m, c the type number of metrics, the collected number of each type metric Metric Anomaly Score: We use the mean µ ik and standard deviation σ ik of the service metrics to calculate service anomaly severity [20][21][22][23][24] . The µ ik is the expected normal value and the σ ik indicates that the metric deviates from the mean.…”

Section: Notation Definitionsmentioning

confidence: 99%

“…Moreover, the current service is influenced by other services if it contains incoming and outcoming abnormal invocations 15,20 . In order to calculate the additional teleportation vector U more accurately in the PageRank algorithm, we calculate the suspicious score by the absolute value of the difference between the numbers of traces that contain incoming abnormal invocations Of iin and outcoming abnormal invocations Of iout .…”

Section: Abnormal Services Rankingmentioning

confidence: 99%

See 1 more Smart Citation

Graph-Based Root Cause Localization in Microservice Systems with Protection Mechanisms

Zhang

Tian

Yang

et al. 2022

Preprint

View full text Add to dashboard Cite

Nowadays, the protection mechanisms are introduced into microservice systems to ensure the stable operation of services. However, existing approaches ignore the impact of protection mechanisms on the root cause localization of abnormal services. Specifically, the circuit breaking and rate limiting mechanisms can refuse service requests and thus change the way of anomaly propagation. Moreover, different service request frequencies and response time make service dependencies change dynamically, resulting in different probabilities of anomaly propagation among services. In this paper, we propose a novel framework named MicroGBPM to locate the root cause of abnormal services, which considers the impact of the protection mechanisms. We model anomaly propagation among services as a dynamically constructed service attributed graph with metrics and traces when a failure occurs. To eliminate the impact of the protection mechanisms, we design a two-stage dynamic calibration strategy to adjust the probability of anomaly propagation among services. Then we propose a random walking approach to calculate the root cause results by using the PageRank algorithm. The experimental results show that MicroGBPM improves the accuracy of root cause localization compared to other approaches in microservice systems with protection mechanisms.

show abstract

“…Prior studies on root cause analysis [25][26][27]30] have mainly focused on a simplified scenario, where the target system is modeled as a single isolated causal graph, and the system's malfunctioning effects can only propagate within the same network of entities. For instance, to identify various types of service root causes, Liu et al [27] generated a service call graph based on domain-specific software and rules.…”

Section: Introductionmentioning

confidence: 99%

Hierarchical Graph Neural Networks for Causal Discovery and Root Cause Localization

Wang¹,

Chen²,

Ni³

et al. 2023

Preprint

View full text Add to dashboard Cite

The goal of root cause analysis is to identify the underlying causes of system problems by discovering and analyzing the causal structure from system monitoring data. It is indispensable for maintaining the stability and robustness of large-scale complex systems. Existing methods mainly focus on the construction of a single effective isolated causal network, whereas many real-world systems are complex and exhibit interdependent structures (i.e., multiple networks of a system are interconnected by cross-network links). In interdependent networks, the malfunctioning effects of problematic system entities can propagate to other networks or different levels of system entities. Consequently, ignoring the interdependency results in suboptimal root cause analysis outcomes.In this paper, we propose REASON, a novel framework that enables the automatic discovery of both intra-level (i.e., withinnetwork) and inter-level (i.e., across-network) causal relationships for root cause localization. REASON consists of Topological Causal Discovery and Individual Causal Discovery. The Topological Causal Discovery component aims to model the fault propagation in order to trace back to the root causes. To achieve this, we propose novel hierarchical graph neural networks to construct interdependent causal networks by modeling both intra-level and inter-level nonlinear causal relations. Based on the learned interdependent causal networks, we then leverage random walk with restarts to model the network propagation of a system fault. The Individual Causal Discovery component focuses on capturing abrupt change patterns of a single system entity. This component examines the temporal patterns of each entity's metric data (i.e., time series), and estimates its likelihood of being a root cause based on the Extreme Value theory. Combining the topological and individual causal scores, the top 𝐾 system entities are identified as root causes. Extensive experiments on three real-world datasets with case studies demonstrate the effectiveness and superiority of the proposed framework.

show abstract

Practical Root Cause Localization for Microservice Systems via Trace Analysis

Cited by 73 publications

References 32 publications

Trace-based microservice anomaly detection through deep learning

Trace-based microservice anomaly detection through deep learning

Graph-Based Root Cause Localization in Microservice Systems with Protection Mechanisms

Hierarchical Graph Neural Networks for Causal Discovery and Root Cause Localization

Contact Info

Product

Resources

About