Understanding and handling alert storm for online service systems

Continuous Incident Triage for Large-Scale Online Service Systems

“…When a failure happens, system monitors will render a large number of incidents to capture different failure symptoms [2]- [4], which can help engineers quickly obtain a big picture of the failure and pinpoint the root cause. For example, "Special instance cannot be migrated" is a critical network failure in Virtual Private Cloud (VPC) service, and the incident "Tunnel † Corresponding author.…”

“…bearing network pack loss" is a signal for this network failure, which is caused by the breakdown of a physical network card on the tunnel path. Due to the large scale and complexity of online service systems, the number of incidents is overwhelming the existing incident management systems [2], [4], [5]. When a service failure occurs, aggregating related incidents can greatly reduce the number of incidents that need to be investigated.…”

“…To aggregate related incidents, one straightforward way is to measure the text similarity between two incident reports [3], [4]. For example, incidents that share similar titles are likely to be related.…”

“…Moreover, the patterns of incidents are collectively influenced by different factors, such as their topological and temporal locality. Existing work [3], [4] combine them by a simple weighted sum, which may not be able to reveal the latent correlations among incidents.…”

“…In this work, we propose GRLIA (stands for Graph Representation Learning-based Incident Aggregation), which is an incident aggregation framework to assist engineers in failure understanding and diagnosis. Different from the existing work of alert storm handling [4] and linked incident identifica-tion [3], we do not rely on incidents' textual similarity. Moreover, we learn incidents' topological and temporal correlations in a unified manner (instead of by a weighted combination).…”