PROUD: PaRallel OUtlier Detection for Streams

Explainable Distance-Based Outlier Detection in Data Streams

“…As far as the NN input type is concerned, we first compute the normalized value of the number of neighbors for each range as shown in Table 1. This procedure yields an array NN of 5 values (for p = 2), where NN [1] holds the normalized number of neighbors for the smallest radius ( R 4 ) and NN [5] corresponds to the biggest range (4R). Finally, the procedure starts checking the array values in descending radius size in order to derive the label.…”

“…For example, Table 2 is transformed to the array outlier_pos = [1, 1, 1, 2, 3]. Then, according to the rules of Table 4, since 1 < outlier_pos [5] ≤ 4, outlier_pos [4] > 1 and outlier_pos [3] = 1, the data point is labeled as part of a sparse micro-cluster. The status-based process is summarized in Algorithm 2.…”

“…PROUD (PaRallel OUtlier Detection for streams) [5] is a distributed framework built on top of Apache Flink that is used for outlier detection jobs on high-volume, intense and continuous streams of data. It is fully modular and encapsulates most of the state-of-the-art single and multi query distance-based outlier detection techniques; furthermore, any new proximity-based technique can easily be incorporated as well.…”

“…The 3-dimensional dataset is called TAO and is a commonly used dataset for streaming outlier detection experiments (e.g., [6]) while the 6-dimensional one is called Mammography. 5 Finally, the 10-dimensional dataset is part of the Forest Cover dataset, which is available at the UCI KDD archive. 6 We have chosen the first 10 dimensions that have a bigger impact on the distance-based outlier detection process.…”

“…3) We apply our methodology in a big-data streaming engine. More specifically, we extend the publicly available PROUD distributed framework [5] by incorporating the solutions as distinct modules in order to provide a fully working and interpretable big-data enabled distance-based outlier detection system. 1 PROUD is build on top of Apache Flink 2 and is a scalable and distributed framework for outlier detection with most of the state-of-the-art distance-based algorithms incorporated along with a self-adaptive partitioning technique.…”