Online Anomaly Detection Under Markov Statistics With Controllable Type-I Error

Neyshabouri

IEEE Trans. Signal Process.

et al. 2019

We introduce an online outlier detection algorithm to detect outliers in a sequentially observed data stream. For this purpose, we use a two-stage filtering and hedging approach. In the first stage, we construct a multimodal probability density function to model the normal samples. In the second stage, given a new observation, we label it as an anomaly if the value of aforementioned density function is below a specified threshold at the newly observed point. In order to construct our multimodal density function, we use an incremental decision tree to construct a set of subspaces of the observation space. We train a single component density function of the exponential family using the observations, which fall inside each subspace represented on the tree. These single component density functions are then adaptively combined to produce our multimodal density function, which is shown to achieve the performance of the best convex combination of the density functions defined on the subspaces. As we observe more samples, our tree grows and produces more subspaces. As a result, our modeling power increases in time, while mitigating overfitting issues. In order to choose our threshold level to label the observations, we use an adaptive thresholding scheme. We show that our adaptive threshold level achieves the performance of the optimal prefixed threshold level, which knows the observation labels in hindsight. Our algorithm provides significant performance improvements over the state of the art in our wide set of experiments involving both synthetic as well as real data.

Section: A Preliminariesmentioning

confidence: 99%

Sequential Outlier Detection Based on Incremental Decision Trees

Neyshabouri

IEEE Trans. Signal Process.

et al. 2019

“…x t is the data present at time t , the complete parameter set of a classical Markov model can be represented by, as shown in [18,22,29,32] :…”

Section: Classical Markov Modelsmentioning

confidence: 99%

“…In this paper, we concentrate on the anomaly detection based on Markov models. Ozkan and Kozat [22] propose an online anomaly detection under Markov statistics with controllable false alarm rate for fast streaming temporal data. This algorithm learns the nominal attributes under possibly varying Markov statistics.…”

Section: Introductionmentioning

confidence: 99%

Anomaly detection based on a dynamic Markov model

Ren

2017

Information Sciences

a b s t r a c tAnomaly detection in sequence data is becoming more and more important in a wide variety of application domains such as credit card fraud detection, health care in medical field, and intrusion detection in cyber security. In the existing anomaly detection approaches, Markov chain techniques are widely accepted for their simple realization and few parameters. However, the short memory property of a classical Markov model ignores the interaction among data, and the long memory property of a higher order Markov model clouds the relationship between the previous data and current test data, and reduces the reliability of the model. Besides, both of these models cannot successfully describe the sequences changing with a tendency. In this paper, we propose an anomaly detection approach based on a dynamic Markov model. This approach segments sequence data by a sliding window. In the sliding window, we define the states of data according to the value of the data and establish a higher order Markov model with a proper order consequently, to balance the length of the memory property and keep up with the trend of sequences. In addition, an anomaly substitution strategy is proposed to prevent the detected anomalies from impacting the building of the models and keep anomaly detection continuously. The experimental results using simulated datasets and real-world datasets have demonstrated that the proposed approach improves the adaptability and stability of anomaly detection in sequence data.

2018 26th Signal Processing and Communications Applications Conference (SIU)

“…Geleneksel yaklaşımları izlemek yerine [1], aykırılık tespiti için UKSB mimarisini kullanan uygun bir objektif fonksiyon tanımlanmış ve bu iyi tanımlanmış fonksiyon ile UKSB mimarisinin parametreleri optimize edilmiştir. Bu bildiride tanımlanan aykırılık tespit algoritması degişken uzunluktaki serileri işleyebilmekte ve zaman serisi verileri için yüksek performans saglayabilmektedir ki zaman serisi verilerinin işlenmesi literatürde oldukça üzerinde durulan bir konudur [6], [7]. Deneylerde, geleneksel metodlara göre [3], [4] yüksek performans artışı gözlenmiştir.…”

Section: Introductionunclassified

A novel anomaly detection approach based on neural networks

Ergen

Kerpiççi

2018

Özetçe-Bu bildiride, etiketsiz çerçevede çalışan ve Uzun Kısa Soluklu Bellek (UKSB) agları temelli olan bir aykırılık sezim algoritması tanıtılmıştır.İlk olarak, degişken uzunluktaki veri dizileri UKSB temelli yapıdan geçirilerek sabit uzunluktaki dizilere dönüştürülmektedir. Sonra, aykırılık sezimi için, Tek Sınıf Destek Vektör Makinası (TS-DVM) algoritmasına dayanan puanlama fonksiyonu tanıtılmaktadır. UKSB yapısı ve TS-DVM formulasyonu için en ideal parametreleri bulmak amacıyla egitim aşamasında kullanılan bayır merkezli algoritma tanıtılmaktadır. Orijinal TS-DVM formulasyonu degiştirildigi için, degiştirilmiş halinin orijinal olana yakınsadıgını gösteren sonuçlar da gösterilmektedir. Tanıtılan algoritma, degişken uzunluktaki veri dizilerini işleyebilmektedir. Ayrıca, zaman serisi verilerinde oldukça yüksek performans göstermektedir. Sayısal örneklerde, geleneksel metodlara göre yüksek bir performans artışı elde edildigi gösterilmektedir. Anahtar Kelimeler-Aykırılık sezimi, destek vektör makinası, zaman serisi verisi, denetlenmeyen yapı, uzun kısa soluklu bellek.