MicroHECL: High-Efficient Root Cause Localization in Large-Scale Microservice Systems

Abstract

Availability issues of industrial microservice systems (e.g., drop of successfully placed orders and processed transactions) directly affect the running of the business. These issues are usually caused by various types of service anomalies which propagate along service dependencies. Accurate and high-efficient root cause localization is thus a critical challenge for large-scale industrial microservice systems. Existing approaches use service dependency graph based analysis techniques to automatically locate root causes. However, these approaches are limited due to their inaccurate detection of service anomalies and inefficient traversing of service dependency graph. In this paper, we propose a high-efficient root cause localization approach for availability issues of microservice systems, called MicroHECL. Based on a dynamically constructed service call graph, MicroHECL analyzes possible anomaly propagation chains, and ranks candidate root causes based on correlation analysis. We combine machine learning and statistical methods and design customized models for the detection of different types of service anomalies (i.e., performance, reliability, traffic). To improve the efficiency, we adopt a pruning strategy to eliminate irrelevant service calls in anomaly propagation chain analysis. Experimental studies show that MicroHECL significantly outperforms two state-of-the-art baseline approaches in terms of both accuracy and efficiency. MicroHECL has been used in Alibaba and achieves a top-3 hit ratio of 68% with root cause localization time reduced from 30 minutes to 5 minutes.

Anomaly Detection and Failure Root Cause Analysis in (Micro)Service-Based Cloud Applications: A Survey

“…Graph-based Analysis. The approach of building and processing topology graphs for determining the possible root causes of performance anomalies is adopted by MonitorRank [38] and MicroHECL [44]. They indeed both rely on services to produce interaction traces, including the start and end times of service interactions, their source and target services, performance metrics (viz., latency, error count, and throughput), and the unique identifier of the corresponding user request.…”

“…This information is then processed to reconstruct service invocation chains for the same user request, which are then combined to build the application topology. They however differ because they consider different time slices for building topologies, and since MonitorRank [38] and MicroHECL [44] exploit topologies to determine the root causes of application-level and servicelevel anomalies, respectively. MonitorRank [38] and MicroHECL [44] also differ in the method applied to enact root cause analysis: MonitorRank [38] performs a random walk on the topology graph, whereas MicroHECL explores it through a breadth-first search (BFS).…”

“…MicroHECL [44] instead builds the topology based on the traces produced in a time window of a given size, which ends at the time when a service experienced the performance anomaly whose root causes are to be determined. Based on such topology, MicroHECL [44] determines the anomaly propagation chains that could have led to the observed anomaly.…”

“…MicroHECL [44] instead builds the topology based on the traces produced in a time window of a given size, which ends at the time when a service experienced the performance anomaly whose root causes are to be determined. Based on such topology, MicroHECL [44] determines the anomaly propagation chains that could have led to the observed anomaly. This is done by starting from the service where the anomaly was observed and by iteratively extending the anomaly propagation chains along the possible anomaly propagation directions, viz., from callee to caller for latency/error count anomalies, and from caller to callee for throughput anomalies.…”

“…When anomaly propagation chains cannot be further extended, the services at the end of each determined propagation chain are considered as possible root causes for the initial anomalous service. MicroHECL [44] finally ranks the possible root causes, based on the Pearson correlation between their performance metrics and those of the service where the performance anomaly was initially observed.…”