A Scalable Approach for Outlier Detection in Edge Streams Using Sketch-based Approximations

Abstract

Dynamic graphs are a powerful way to model an evolving set of objects and their ongoing interactions. A broad spectrum of systems, such as information, communication, and social, are naturally represented by dynamic graphs. Outlier (or anomaly) detection in dynamic graphs can provide unique insights into the relationships of objects and identify novel or emerging relationships. To date, outlier detection in dynamic graphs has been studied in the context of graph streams, focusing on the analysis and comparison of entire graph objects. However, the volume and velocity of data are necessitating a transition from outlier detection in the context of graph streams to outlier detection in the context of edge streams-where the stream consists of individual graph edges instead of entire graph objects.In this paper, we propose the first approach for outlier detection in edge streams. We first describe a high-level model for outlier detection based on global and local structural properties of a stream. We then propose a novel application of the Count-Min sketch for approximating these properties, and prove probabilistic error bounds on our edge outlier scoring functions. Our sketch-based implementation provides a scalable solution, having constant time updates and constant space requirements. Experiments on synthetic and real-world datasets demonstrate our method's scalability, effectiveness for discovering outliers, and the effects of approximation.

AddGraph: Anomaly Detection in Dynamic Graph Using Attention-based Temporal GCN

“…It builds a generative model for edges in a node cluster, and the model can also be used to produce anomalous score for a given edge. • CM-Sketch [Ranshous et al, 2016]. It uses the local structural feature and historical behavior near an edge to measure whether the edge is anomalous or not.…”

“…Besides the structural features, the temporal ones are considered in the anomaly detection. CM-Sketch [Ranshous et al, 2016] is a sketch-based method, which uses the local structural information and historical behavior near an edge to decide whether the edge is anomalous or not. Spot-Light [Eswaran et al, 2018] randomly samples a series of node sets from the entire node set, and encodes the the graph at each timestamp to a vector by computing the overlap between these sets and the nodes of current edge set.…”

“…In order to achieve the goal quickly, anomalous users usually control multiple accounts to perform these operations in a short time period. The anomaly detection in dynamic graph, especially anomalous edges detection, is then highly needed before the data are fed into the following tasks [Akoglu et al, 2015;Ranshous et al, 2015].…”

SpotLight

“…(iv) Anomalous edge detection approaches [2,17,22]: The first two methods score the likelihood of an edge based on the community structure [2], prior occurrence preferential attachment and homophily information [22]. By scoring edges independent of each other, these methods miss complex structural (e.g., dense subgraph) anomalies.…”

“…Examples include transportation logs (w cabs travel from location s to location d), network communication logs (w packets sent by IP address s to We consider the problem of near real-time anomaly detection in such settings. Due to the fluid nature of what is considered 'normal', prior works typically focus on detecting specific anomalous changes to the graph, e.g., bridge edges [22,26], hotspot nodes [31], changes to community structure [27,28], graph metrics [8,10], etc. In this work, we focus on detecting anomalies involving the sudden appearance or disappearance of a large dense directed subgraphs (near bicliques), which is useful in numerous applications: detecting attacks (port scan, denial of service) in network communication logs, interesting/fraudulent behavior creating spikes of activity in user-user communication logs (scammers who operate fast and in bulk), important events (holidays, large delays) creating abnormal traffic in/out flow to certain locations, etc.…”