Fast, Accurate, and Flexible Algorithms for Dense Subtensor Mining

Abstract

How can we detect fraudulent lockstep behavior in large-scale multi-aspect data (i.e., tensors)? Can we detect it when data are too large to fit in memory or even on a disk? Past studies have shown that dense subtensors in real-world tensors (e.g., social media, Wikipedia, TCP dumps, etc.) signal anomalous or fraudulent behavior such as retweet boosting, bot activities, and network attacks. Thus, various approaches, including tensor decomposition and search, have been proposed for detecting dense subtensors rapidly and accurately. However, existing methods have low accuracy, or they assume that tensors are small enough to fit in main memory, which is unrealistic in many real-world applications such as social media and web. To overcome these limitations, we propose D-CUBE, a disk-based dense-subtensor detection method, which also can run in a distributed manner across multiple machines. Compared to state-of-the-art methods, D-CUBE is (1) Memory Efficient: requires up to 1,600× less memory and handles 1,000× larger data (2.6TB), (2) Fast: up to 7× faster due to its near-linear scalability, (3) Provably Accurate: gives a guarantee on the densities of the detected subtensors, and (4) Effective: spotted network attacks from TCP dumps and synchronized behavior in rating data most accurately.

No Place to Hide: Catching Fraudulent Entities in Tensors

“…In contrast to adding feature values to seed blocks, M-Zoom [30] removes feature values from the initial tensor one by one using a similar greedy strategy, providing a 1/N -approximation guarantee for finding the optimum (where N is the number of dimensions in the tensor). M-Biz [31] also starts from a seed block and then greedily adds or removes feature values until the block reaches a local optimum. Unlike M-Zoom, D-Cube [32] deletes a set of feature values on each step to reduce the number of iterations, and is implemented in a distributed disk-based manner.…”

“…D-Cube provides the same approximation guarantee as M-Zoom. MAF [20] CrossSpot [12] M-zoom [30] M-biz [31] D-cube [32] ISG+D-Spot Applicable to N-dimensional data?…”

“…Other density metrics listed in [32] are also effective for fraud detection. It is broadly true that all density measures are functions of the cardinalities of the dimensions and masses of B and R. In R, previous studies [12,[30][31][32] have focused on detecting the top-k densest blocks in terms of a density metric. In the remainder of this paper, we use the density metric ρ to illustrate our key points.…”

“…Thus, catching hidden-densest blocks has a significant utility in the real world. Unfortunately, the problem is intractable using existing approaches [12,[30][31][32].…”

“…when N is sufficiently large. Assuming ρ(B, [N − 1]) is very low, then the methods in [12,[30][31][32], which try to find the block B that maximizes ρ(B, [N ]), have a limited ability to detect the hiddendensest block. Second, consider a block B formed by fraudulent entities, in which B is only the densest on {A 2 , A 3 , A 5 }, and thus ρ(B, {2, 3, 5}) is maximal.…”