Designing a Streaming Algorithm for Outlier Detection in Data Mining—An Incrementa Approach

Yu, Kangqing; Shi, Wei; Santoro, Nicola

doi:10.3390/s20051261

Cited by 29 publications

(23 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, the accuracy and reliability of the microcirculatory functional values are two important determinants of the common microcirculatory framework. Hence, it is necessary to pre-process outliers for the input data set 30 . Comparing various outlier processing algorithms (missing values, mean correction, etc.)…”

Section: Discussionmentioning

confidence: 99%

Common Microcirculatory Framework for Monitoring Integrated Microcirculation

Wang

et al. 2021

Preprint

View full text Add to dashboard Cite

Wide variation in magnitudes, units, and ranges of the microcirculatory variables brings hindrance in describing and evaluating the integrated microcirculatory function of tissues. We designed to establish common microcirculatory framework that contains microhemodynamic and microcirculatory oxygen parameters. To integrate microcirculatory information, demo microcirculatory permutations were generated by a computer algorithm based on microcirculatory characteristics. Four dimensionless methods (Z-score, Min-max, L2, and median scaling) were applied to transform microcirculatory data set into the dimensionless form. Three-dimensional (3-D) common microcirculatory framework was constructed and visualized by using Python and Apache ECharts. The performance of the four dimensionless methods in the pre-processing of multiple microcirculatory variables and the establishment of the common microcirculatory framework were compared. Microhemodynamic and microcirculatory oxygen parameters were embedded in the common microcirculatory framework. After processing by Min-max normalization, the transformed multiple microcirculatory values remained positive with fixed range mapping within [0, 1] and maintained the identity property of microcirculation both of microhemodynamic and microcirculatory oxygen variables in the common microcirculatory framework. Conclusively, Min-max normalization displays preferable integration efficiency, compatibility, and adaptability in the establishment of the 3-D visualized multiparametric common microcirculatory framework.

show abstract

Section: Discussionmentioning

confidence: 99%

Common Microcirculatory Framework for Monitoring Integrated Microcirculation

Wang

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The output is the update of each CMC and WMC. The pseudocode starts with the energy of all CMCs with the value of 1 Decay . Afterwards, for each T in CMCs, the energy is checked whenever the energy is lower than 0, and T will be removed from all CMCs and added to WMCs.…”

Section: Boceds Moving Weak Micro-clusters To a Buffermentioning

confidence: 99%

“…Data streams are a continuous, infinite series of data records followed and arranged by embedded or precise timestamps [1]. With sensors becoming prevalent in humans' daily lives, it's clear that the availability of data streams is exponentially increasing [2].…”

Section: Introductionmentioning

confidence: 99%

A Clustering Algorithm for Evolving Data Streams Using Temporal Spatial Hyper Cube

et al. 2022

View full text Add to dashboard Cite

As applications generate massive amounts of data streams, the requirement for ways to analyze and cluster this data has become a critical field of research for knowledge discovery. Data stream clustering’s primary objective and goal are to acquire insights into incoming data. Recognizing all possible patterns in data streams that enter at variable rates and structures and evolve over time is critical for acquiring insights. Analyzing the data stream has been one of the vital research areas due to the inevitable evolving aspect of the data stream and its vast application domains. Existing algorithms for handling data stream clustering consider adding various data summarization structures starting from grid projection and ending with buffers of Core-Micro and Macro clusters. However, it is found that the static assumption of the data summarization impacts the quality of clustering. To fill this gap, an online clustering algorithm for handling evolving data streams using a tempo-spatial hyper cube called BOCEDS TSHC has been developed in this research. The role of the tempo-spatial hyper cube (TSHC) is to add more dimensions to the data summarization for more degree of freedom. TSHC when added to Buffer-based Online Clustering for Evolving Data Stream (BOCEDS) results in a superior evolving data stream clustering algorithm. Evaluation based on both the real world and synthetic datasets has proven the superiority of the developed BOCEDS TSHC clustering algorithm over the baseline algorithms with respect to most of the clustering metrics.

show abstract

“…Widely accepted and popular solutions, such as Hoeffding Trees [14] or Online Random Forests [15], achieve good accuracy and robustness in data streams [16] but are not designed to operate on unlabeled data. Over the past couple of years, methods have been proposed that satisfy the unsupervised and online requirement, such as [17][18][19], but just a few, Isolation Forest (iForest) [20], HS-Trees [21], RS-Hash [22] and Loda [23], have been shown to outperform numerous competitors and are therefore regarded as state of the art [24,25]. Even if iForest was originally intended as an offline algorithm, a handful of variants, such as [16,[26][27][28][29], have been proposed that are adapting it or are taking advantage of its concept to operate on SD.…”

Section: Related Work 21 Aspects On Unsupervised Online Outlier Detectionmentioning

confidence: 99%

Exploiting the Outcome of Outlier Detection for Novel Attack Pattern Recognition on Streaming Data

et al. 2021

View full text Add to dashboard Cite

Future-oriented networking infrastructures are characterized by highly dynamic Streaming Data (SD) whose volume, speed and number of dimensions increased significantly over the past couple of years, energized by trends such as Software-Defined Networking or Artificial Intelligence. As an essential core component of network security, Intrusion Detection Systems (IDS) help to uncover malicious activity. In particular, consecutively applied alert correlation methods can aid in mining attack patterns based on the alerts generated by IDS. However, most of the existing methods lack the functionality to deal with SD data affected by the phenomenon called concept drift and are mainly designed to operate on the output from signature-based IDS. Although unsupervised Outlier Detection (OD) methods have the ability to detect yet unknown attacks, most of the alert correlation methods cannot handle the outcome of such anomaly-based IDS. In this paper, we introduce a novel framework called Streaming Outlier Analysis and Attack Pattern Recognition, denoted as SOAAPR, which is able to process the output of various online unsupervised OD methods in a streaming fashion to extract information about novel attack patterns. Three different privacy-preserving, fingerprint-like signatures are computed from the clustered set of correlated alerts by SOAAPR, which characterizes and represents the potential attack scenarios with respect to their communication relations, their manifestation in the data's features and their temporal behavior. Beyond the recognition of known attacks, comparing derived signatures, they can be leveraged to find similarities between yet unknown and novel attack patterns. The evaluation, which is split into two parts, takes advantage of attack scenarios from the widely-used and popular CICIDS2017 and CSE‐CIC‐IDS2018 datasets. Firstly, the streaming alert correlation capability is evaluated on CICIDS2017 and compared to a state-of-the-art offline algorithm, called Graph-based Alert Correlation (GAC), which has the potential to deal with the outcome of anomaly-based IDS. Secondly, the three types of signatures are computed from attack scenarios in the datasets and compared to each other. The discussion of results, on the one hand, shows that SOAAPR can compete with GAC in terms of alert correlation capability leveraging four different metrics and outperforms it significantly in terms of processing time by an average factor of 70 in 11 attack scenarios. On the other hand, in most cases, all three types of signatures seem to reliably characterize attack scenarios such that similar ones are grouped together, with up to 99.05\% similarity between the FTP and SSH Patator attack.intrusion detection; alert analysis; alert correlation; outlier detection; attack scenario; streaming data; network security

show abstract

Designing a Streaming Algorithm for Outlier Detection in Data Mining—An Incrementa Approach

Cited by 29 publications

References 41 publications

Common Microcirculatory Framework for Monitoring Integrated Microcirculation

Common Microcirculatory Framework for Monitoring Integrated Microcirculation

A Clustering Algorithm for Evolving Data Streams Using Temporal Spatial Hyper Cube

Exploiting the Outcome of Outlier Detection for Novel Attack Pattern Recognition on Streaming Data

Contact Info

Product

Resources

About