NetSpot: Spotting Significant Anomalous Regions on Dynamic Networks

Abstract

How to spot and summarize anomalies in dynamic networks such as road networks, communication networks and social networks? An anomalous event, such as a traffic accident, a denial of service attack or a chemical spill, can affect several near-by edges and make them behave abnormally, over several consecutive time-ticks. We focus on spotting and summarizing such significant anomalous regions, spanning space (i.e. nearby edges), as well as time.Our first contribution is the problem formulation, namely finding all such Significant Anomalous Regions (SAR). The next contribution is the design of novel algorithms: an expensive, exhaustive algorithm, as well as an efficient approximation, called NetSpot. Compared to the exhaustive algorithm, NetSpot is up to one order of magnitude faster in real data, while achieving less than 4% average relative error rate. In synthetic datasets, it is more than 30 times faster and solves large problem instances that are otherwise infeasible. The final contribution is the validation on real data: we demonstrate the utility of NetSpot for inferring accidents on road networks and detecting patterns of anomalous access to subnetworks of Wikipedia. We also study NetSpot's scalability in large social, transportation and synthetic evolving networks, spanning in total up to 50 million edges.

Near-Optimal and Practical Algorithms for Graph Scan Statistics

“…is the posterior probability of seeing n t e interactions given the counts in the previous days, and µ = 0.05 is a significance threshold. This weighing function [14] is positiveincreasing if the posterior probability is less than µ and negative-decreasing otherwise. After assigning weights to edges, we solve the Heaviest Dynamic Subgraph problem [4] using algorithm ColCodeNW (See Section 5.3.3).…”

Intrusion detection in network flows based on an optimized clustering criterion

“…Many recent research studies have also employed these methods in order to detect the behavior of the network and whether it is a normal or malicious case [12,27].…”

Efficient and flexible algorithms for monitoring distance-based outliers over data streams

“…Outlier elimination from aggregate query results by discovering appropriate predicates is proposed in [16]. Recent trends in outlier detection include the investigation of techniques that are particularly significant for big data analysis, such as dealing with high dimensionality, e.g., [17,18], and considering graph-based data types common in social networks, e.g., [19,20], categorical data [21] and video streams [22]. Apart from the fact that outliers are important in many applications, their discovery allows the data set to be "cleaned" to apply a particular model [23], while in many cases, their detection is a by-product of clustering, e.g., as in [24].…”