A Novel Framework for Context-aware Outlier Detection in Big Data Streams

Abstract

Outlier and anomaly detection has always been a critical problem in many fields. Although it has been investigated deeply in data mining, the problem has become more difficult and critical in the Big Data era since the volume, velocity and variety of data change drastically with rather complicated types of outliers. In such an environment, where real-time outlier detection and analysis over data streams is a necessity, the existing solutions are no longer effective and sufficient. While many existing algorithms and approaches consider the content of the data stream, there are few approaches which consider the context and conditions in which the content has been produced. In this paper, we propose a novel framework for contextual outlier detection in big data streams which inject the contextual attributes in the stream content as a primary input for outlier detection rather than using the stream content alone or applying the contextual detection on content anomalies only. The detection algorithm incorporates two approaches; the first, a supervised detection method and the other, an unsupervised, which allows the detection process to adapt to the normal change in the stream behavior over time. The detected outliers are either both content and contextual outliers or contextual outliers only. The proposed contextual detection approach prunes the false positive outliers and detects the true negative outliers at the same time. Moreover, in this framework, the detection engine preserves both outliers and context values in which those outliers were detected to be used in the engine self-training and in outliers modeling in order to enhance the outlier prediction accuracy.

The discrete Fourier transformation for seasonality and anomaly detection of an application to rare data

“…Additionally, a new method for detecting anomalies in binary, rare data is presented: the sum of distances. Previous research has employed various methods to find and address anomalies in non-rare and non-binary seasonal data (Ahmad and Dowaji, 2018; Kassimi et al , 2018; Borel et al , 2012; Wang et al , 2015), but these techniques are not applicable to the data used in this paper. Using the sum of distances method, expected events which have not occurred and unexpected events which have occurred at various sampling frequencies can be detected.…”

“…A number of prior works have aimed to address anomalies in time signals. Supervised and unsupervised methods have been employed to discover global, local and hidden anomalies (Ahmad and Dowaji, 2018), (Kassimi et al , 2018). Previously, anomaly detection in time series has included range-invariant anomaly detection (Borel et al , 2012), Fourier phase spectrums (Wang et al , 2015) and Fourier phase reconstruction (Hung et al , 2016).…”

“…While all of these works perform analysis on seasonal data, they do not work with binary or rare event signals. Techniques such as multivariate normal distribution scoring (Ahmad and Dowaji, 2018) are not applicable when the data is rare and binary because traditional methods, such as outlier detection (Lakhina et al , 2004), do not work.…”