Robust anomaly detection algorithms for real-time big data: Comparison of algorithms

Hasani, Zirije

doi:10.1109/meco.2017.7977130

Cited by 23 publications

(18 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our proposed algorithm (HW-GA) [1] with GA optimized parameters (α, β, γ, δ, k, n) and with improved is copared with ARIMA, MA (implemented in our previous work [2]), HW where smoothing parameters are calculated by formula and default MASE (HW calc.MASE), HW by default smoothing parameters (optimized in R) and default MASE (HW def.MASE), HW by default smoothing parameters and improved (HW def.MASE(k,n)).…”

Section: Resultsmentioning

confidence: 99%

“…In our previous research [2] we have compared many algorithms as MAD, RunMAD, Boxplot, Twitter ADVec, DBSCAN, Moving Range Technique, Statistical Control Chart Techniques, ARIMA and Moving Average, to nd which one is faster. So the most important aspects that we considered, in order to nd anomaly detection algorithm suitable for future implementation in the online environment was the execution time (complexity), the CPU usage and the satisfactory quality of algorithm (measured through…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Performance Comparison of Anomaly Detection Algorithms for Streaming Data

Hasani

Fondaj

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Most of the today's world data are streaming, time-series data, where anomalies detection gives significant information of possible critical situations. Yet, detecting anomalies in big streaming data is a difficult task, requiring detectors to acquire and process data in a real-time, as they occur, even before they are stored and instantly alarm on potential threats. Suitable to the need for real-time alarm and unsupervised procedures for massive streaming data anomaly detection, algorithms have to be robust, with low processing time, eventually at the cost of the accuracy. In this work we compare the performance of our proposed anomaly detection algorithm HW-GA[1] with other existing methods as ARIMA [10], Moving Average [11] and Holt Winters [12]. The algorithms are tested and results are visualized in the system R, on the three Numenta datasets, with known anomalies and own e-dnevnik dataset with unknown anomalies. Evaluation is done by comparing achieved results (the algorithm execution time and CPU usage). Our interest is monitoring of the streaming log data that are generating in the national educational network (e-dnevnik) that acquires a massive number of online queries and to detect anomalies in order to scale up performance, prevent network downs, alarm on possible attacks and similar.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Performance Comparison of Anomaly Detection Algorithms for Streaming Data

Hasani

Fondaj

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The testing is done in real-time data that comes from the e-dnevnik application. This tool enables us to implement our evaluated algorithms (Hasani, 2017) and visualize the results. Visualization of the real-time anomaly detected is an important part of the infrastructure as a powerful tool for online monitoring the work of the system.…”

Section: Resultsmentioning

confidence: 99%

“…As part of this infrastructure, Timelion, enables programming and implementation of user defined anomaly detection algorithms, as example algorithms we analyzed in (Hasani, 2017). There are some algorithms used for anomaly detection and we explore some of them and implement in real time environment in our proposed infrastructure.…”

Section: Anomaly Detection With Timelionmentioning

confidence: 99%

Real Time Anomaly Detection in Massive Data Streams with ELK Stack

Fondaj¹,

Hasani²

2019

Journal of Computer Science

Self Cite

View full text Add to dashboard Cite

Real time anomaly detection is very popular topic nowadays this because the number of data generated every day is larger and larger. Facing with the phenomena of Big Data is not an easy task. The main aim of this research is to fine appropriate architecture for real-time big data analytic and its main task is to detect anomalies in this real-time data. In this paper we show the implementation of anomaly detection algorithm in real time infrastructure in order to find anomalies as soon as possible. We have proposed architecture for real time anomaly detection by adding some new components and the main part of the infrastructure is Timelion which enable implementation of different algorithms for anomaly detection. The research is focused to develop infrastructure to monitor ednevnik (education national system in Macedonia) application server and to detect errors in order to scale up the performance.

show abstract

“…is compared with ARIMA, MA (implemented in our previous work[54]), HTM[5] algorithm, HW where smoothing parameters are calculated by formula and default MASE (HW calc. MASE), HW by default smoothing parameters (optimized in R) and default MASE (HW def.…”

mentioning

confidence: 99%

Survey and Proposal of an Adaptive Anomaly Detection Algorithm for Periodic Data Streams

Hasani¹,

Krrabaj²

2019

JCC

Self Cite

View full text Add to dashboard Cite

Real-time anomaly detection of massive data streams is an important research topic nowadays due to the fact that a lot of data is generated in continuous temporal processes. There is a broad research area, covering mathematical, statistical, information theory methodologies for anomaly detection. It addresses various problems in a lot of domains such as health, education, finance, government, etc. In this paper, we analyze the state-of-the-art of data streams anomaly detection techniques and algorithms for anomaly detection in data streams (time series data). Critically surveying the techniques' performances under the challenge of real-time anomaly detection of massive high-velocity streams, we conclude that the modeling of the normal behavior of the stream is a suitable approach. We evaluate Holt-Winters (HW), Taylor's Double Holt-Winters (TDHW), Hierarchical temporal memory (HTM), Moving Average (MA), Autoregressive integrated moving average (ARIMA) forecasting models, etc. Holt-Winters (HW) and Taylor's Double Holt-Winters (TDHW) forecasting models are used to predict the normal behavior of the periodic streams, and to detect anomalies when the deviations of observed and predicted values exceeded some predefined measures. In this work, we propose an enhancement of this approach and give a short description about the algorithms and then they are categorized by type of prediction as: predictive and non-predictive algorithms. We implement the Genetic Algorithm (GA) to periodically optimize HW and TDHW smoothing parameters in addition to the two sliding windows parameters that improve Hyndman's MASE measure of deviation, and value of the threshold parameter that defines no anomaly confidence interval [1]. We also propose a new optimization function based on the input training datasets with the annotated anomaly intervals, in order to detect the right anomalies and minimize the number of false ones. The proposed method is evaluated on the known anomaly detection benchmarks NUMENTA and Yahoo datasets with annotated anomalies and real log data generated by the National education information system

show abstract

Robust anomaly detection algorithms for real-time big data: Comparison of algorithms

Cited by 23 publications

References 4 publications

Performance Comparison of Anomaly Detection Algorithms for Streaming Data

Performance Comparison of Anomaly Detection Algorithms for Streaming Data

Real Time Anomaly Detection in Massive Data Streams with ELK Stack

Survey and Proposal of an Adaptive Anomaly Detection Algorithm for Periodic Data Streams

Contact Info

Product

Resources

About