Online anomaly detection for multi‐source VMware using a distributed streaming framework

Solaimani, Mohiuddin; Iftekhar, Mohammed; Khan, Latifur; Thuraisingham, Bhavani; Ingram, Joe; Şeker, Şadi Evren

doi:10.1002/spe.2390

Cited by 8 publications

(10 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, Flume agent is capable of creating various channels to process the incoming data and concurrently store raw data for further analysis 2 . The availability will be enhanced by the multiple channel selection methods and at the same time the network overhead will be minimized 3 …”

Section: Proposed Frameworkmentioning

confidence: 99%

“…2 The availability will be enhanced by the multiple channel selection methods and at the same time the network overhead will be minimized. 3 In this work, the Flume has been used to read the log files that are being generated in real time by BroIDS. Moreover, some of the components in the Flume agent have been redesigned to introduce the load balancing using master node.…”

Section: Flumementioning

confidence: 99%

“…Moreover, some of the components in the Flume agent have been redesigned to introduce the load balancing using master node. 3 This modification enables each collector node to fully utilize the memory for process and this strategy plays vital role to monitor the usage of memory in real time. Lastly, Flume is used to write the data being read into a Kafka topic for further process.…”

Section: Flumementioning

confidence: 99%

“…Furthermore, it delivers better queuing real-time data streams system with reliable, high throughput, and low latency. 3 The data stream from Kafka topics can be received by distributed analytic platform for further processing and implementing diverse analytics capabilities. 1 Furthermore, most of the real-time systems attain various benefits such as developing a message passing system to collect information for processing and storing the event in real time.…”

Section: Kafkamentioning

confidence: 99%

“…However, existing real-time anomaly detection approach is challenged by the magnitude of threats, volume, and velocity of the data. [2][3][4][5] Given these challenges, current anomaly detection techniques are often characterized by higher execution time and utilization of bulk memory, which yields inadequate and inaccurate anomaly detection in real-time. Even though storing, processing, and analyzing the incoming data offer no value in on-the-spot remedial actions, it may help to explain the threat, and the outcome would act as a feedback for the future detections.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Clustering‐based real‐time anomaly detection—A breakthrough in big data technologies

Habeeb

Nasaruddin

Gani

et al. 2019

Trans Emerging Tel Tech

View full text Add to dashboard Cite

Off late, the ever increasing usage of a connected Internet‐of‐Things devices has consequently augmented the volume of real‐time network data with high velocity. At the same time, threats on networks become inevitable; hence, identifying anomalies in real time network data has become crucial. To date, most of the existing anomaly detection approaches focus mainly on machine learning techniques for batch processing. Meanwhile, detection approaches which focus on the real‐time analytics somehow deficient in its detection accuracy while consuming higher memory and longer execution time. As such, this paper proposes a novel framework which focuses on real‐time anomaly detection based on big data technologies. In addition, this paper has also developed streaming sliding window local outlier factor coreset clustering algorithms (SSWLOFCC), which was then implemented into the framework. The proposed framework that comprises BroIDS, Flume, Kafka, Spark streaming, SparkMLlib, Matplot and HBase was evaluated to substantiate its efficacy, particularly in terms of accuracy, memory consumption, and execution time. The evaluation is done by performing critical comparative analysis using existing approaches, such as K‐means, hierarchical density‐based spatial clustering of applications with noise (HDBSCAN), isolation forest, spectral clustering and agglomerative clustering. Moreover, Adjusted Rand Index and memory profiler package were used for the evaluation of the proposed framework against the existing approaches. The outcome of the evaluation has substantially proven the efficacy of the proposed framework with a much higher accuracy rate of 96.51% when compared to other algorithms. Besides, the proposed framework also outperformed the existing algorithms in terms of lesser memory consumption and execution time. Ultimately the proposed solution enable analysts to precisely track and detect anomalies in real time.

show abstract

Section: Proposed Frameworkmentioning

confidence: 99%

Section: Flumementioning

confidence: 99%

Section: Flumementioning

confidence: 99%

Section: Kafkamentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Clustering‐based real‐time anomaly detection—A breakthrough in big data technologies

Habeeb

Nasaruddin

Gani

et al. 2019

Trans Emerging Tel Tech

View full text Add to dashboard Cite

show abstract

Dynamics signature based anomaly detection

Goenawan

et al. 2021

Softw Pract Exp

View full text Add to dashboard Cite

Identifying anomalies, especially weak anomalies in constantly changing targets, is more difficult than in stable targets. In this article, we borrow the dynamics metrics and propose the concept of dynamics signature (DS) in multi‐dimensional feature space to efficiently distinguish the abnormal event from the normal behaviors of a variable star. The corresponding dynamics criterion is proposed to check whether a star's current state is an anomaly. Based on the proposed concept of DS, we develop a highly optimized DS algorithm that can automatically detect anomalies from millions of stars' high cadence sky survey data in real‐time. Microlensing, which is a typical anomaly in astronomical observation, is used to evaluate the proposed DS algorithm. Two datasets, parameterized sinusoidal dataset containing 262,440 light curves and real variable stars based dataset containing 462,996 light curves are used to evaluate the practical performance of the proposed DS algorithm. Experimental results show that our DS algorithm is highly accurate, sensitive to detecting weak microlensing events at very early stages, and fast enough to process 176,000 stars in less than 1 s on a commodity computer.

show abstract