Pattern-Based Anomaly Detection in Mixed-Type Time Series

Feremans, Len; Vercruyssen, Vincent; Čule, Boris; Meert, Wannes; Goethals, Bart

doi:10.1007/978-3-030-46150-8_15

Cited by 31 publications

(37 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Roughly speaking, it is a recurrent neural network, yet with a more complex neuron structure and an unsupervised learning rule. Other algorithms reported in [ 81 ] have weaker results, as well as the reports in [ 84 ] on the same data. Table 4 summarizes these results.…”

Section: Results and Comparisons On Benchmark Trip Record Datasupporting

confidence: 58%

Anomaly Detection for Individual Sequences with Applications in Identifying Malicious Tools

Siboni

Cohen

2020

Entropy

View full text Add to dashboard Cite

Anomaly detection refers to the problem of identifying abnormal behaviour within a set of measurements. In many cases, one has some statistical model for normal data, and wishes to identify whether new data fit the model or not. However, in others, while there are normal data to learn from, there is no statistical model for this data, and there is no structured parameter set to estimate. Thus, one is forced to assume an individual sequences setup, where there is no given model or any guarantee that such a model exists. In this work, we propose a universal anomaly detection algorithm for one-dimensional time series that is able to learn the normal behaviour of systems and alert for abnormalities, without assuming anything on the normal data, or anything on the anomalies. The suggested method utilizes new information measures that were derived from the Lempel–Ziv (LZ) compression algorithm in order to optimally and efficiently learn the normal behaviour (during learning), and then estimate the likelihood of new data (during operation) and classify it accordingly. We apply the algorithm to key problems in computer security, as well as a benchmark anomaly detection data set, all using simple, single-feature time-indexed data. The first is detecting Botnets Command and Control (C&C) channels without deep inspection. We then apply it to the problems of malicious tools detection via system calls monitoring and data leakage identification.We conclude with the New York City (NYC) taxi data. Finally, while using information theoretic tools, we show that an attacker’s attempt to maliciously fool the detection system by trying to generate normal data is bound to fail, either due to a high probability of error or because of the need for huge amounts of resources.

show abstract

Section: Results and Comparisons On Benchmark Trip Record Datasupporting

confidence: 58%

Anomaly Detection for Individual Sequences with Applications in Identifying Malicious Tools

Siboni

Cohen

2020

Entropy

View full text Add to dashboard Cite

show abstract

“…Matrix profile is a successful technique in unsupervised rare pattern-based time-series anomaly detection [40]. It was developed based on the nearest neighbour algorithm.…”

Section: Discussionmentioning

confidence: 99%

A Two-Dimensional Sparse Matrix Profile DenseNet for COVID-19 Diagnosis Using Chest CT Images

Liu

Leung

2020

IEEE Access

View full text Add to dashboard Cite

COVID-19 is a newly identified disease, which is very contagious and has been rapidly spreading across different countries around the world, calling for rapid and accurate diagnosis tools. Chest CT imaging has been widely used in clinical practice for disease diagnosis, but image reading is still a time-consuming work. We aim to integrate an image preprocessing technology for anomaly detection with supervised deep learning for chest CT imaging-based COVID-19 diagnosis. In this study, a matrix profile technique was introduced to CT image anomaly detection in two levels. At one-dimensional level, CT images were simply flatted and transformed to a one-dimensional vector so that the matrix profile algorithm could be implemented for them directly. At two-dimensional level,a matrix profile was calculated in a sliding window way for every segment in the image. An anomaly severity score (CT-SS) was calculated, and the difference of the CT-SS between the COVID-19 CT images and Non-COVID-19 CT images was tested. A sparse anomaly mask was calculated and applied to penalize the pixel values of each image. The anomaly weighted images were then used to train standard DenseNet deep learning models to distinguish the COVID-19 CT from Non-COVID-19 CT images. A VGG19 model was used as a baseline model for comparison. Although extra finetuning needs to be done manually, the onedimensional matrix profile method could identify the anomalies successfully. Using the two-dimensional matrix profiling method, CT-SS and anomaly weighted image can be successfully generated for each image. The CT-SS significantly differed among the COVID-19 CT images and Non-COVID-19 CT images (p − value < 0.05). Furthermore, we identified a potential causal association between the number of underlying diseases of a COVID-19 patient and the severity of the disease through statistical mediation analysis. Compared to the raw images, the anomaly weighted images showed generally better performance in training the DenseNet models with different architectures for diagnosing COVID-19, which was validated using two publicly available COVID-19 lung CT image datasets. The metric Area Under the Curve(AUC) on one dataset were 0.7799(weighted)vs. 0.7391(unweighted), 0.7812(weighted) vs. 0.7410(unweighted), 0.7780(weighted) vs. 0.7399(unweighted), 0.7045(weighted) vs. 0.6910(unweighted) for DenseNet121, DenseNet169, DenseNet201, and the baseline model VGG19, respectively. The same trend was observed using another independent dataset. The significant results revealed the critical value of using this existing state-of-the-art algorithm for image anomaly detection. Furthermore, the end-to-end model structure has the potential to work as a rapid tool for clinical imaging-based diagnosis. INDEX TERMS Rare pattern mining, matrix profile, COVID-19 CT images, risk score, DenseNet, mediation analysis I. INTRODUCTION U NSUPERVISED anomaly detection using rare pattern mining is one of the most intuitive medical imaging based disease diagnosis methods [1]. People without medical expe...

show abstract

“…LOF has shown great success in dealing with various industrial IoT applications such as manufacturing [18], intelligent transportation [12], and among others. However, solutions to urban traffic anomaly detection [23,36,44] are only able to identify local outliers, where global outliers may be identified from urban traffic data. These solutions are also lacking privacy, where they do not provide a secure mechanism for the distributed analysis process.…”

Section: Introductionmentioning

confidence: 99%

SS-ITS: secure scalable intelligent transportation systems

et al. 2021

View full text Add to dashboard Cite

This paper introduces a secure and scalable intelligent transportation and human behavior system to accurately discover knowledge from urban traffic data. The data are secured using blockchain learning technology, where the scalability is ensured by a threaded GPU. In addition, different optimizations are provided to efficiently process data on the GPU. A reinforcement deep learning algorithm is also established to merge local knowledge discovered on each site into global knowledge. To demonstrate the applicability of the proposed framework, intensive experiments have been carried out on well-known intelligent transportation and human behavior data. Our results show that our proposed framework outperforms the baseline solutions for the outlier detection use case.

show abstract

Pattern-Based Anomaly Detection in Mixed-Type Time Series

Cited by 31 publications

References 42 publications

Anomaly Detection for Individual Sequences with Applications in Identifying Malicious Tools

Anomaly Detection for Individual Sequences with Applications in Identifying Malicious Tools

A Two-Dimensional Sparse Matrix Profile DenseNet for COVID-19 Diagnosis Using Chest CT Images

SS-ITS: secure scalable intelligent transportation systems

Contact Info

Product

Resources

About