Real-Time Detection of Global Cyberthreat Based on Darknet by Estimating Anomalous Synchronization Using Graphical Lasso

Abstract

With the rapid evolution and increase of cyberthreats in recent years, it is necessary to detect and understand it promptly and precisely to reduce the impact of cyberthreats. A darknet, which is an unused IP address space, has a high signal-to-noise ratio, so it is easier to understand the global tendency of malicious traffic in cyberspace than other observation networks. In this paper, we aim to capture global cyberthreats in real time. Since multiple hosts infected with similar malware tend to perform similar behavior, we propose a system that estimates a degree of synchronizations from the patterns of packet transmission time among the source hosts observed in unit time of the darknet and detects anomalies in real time. In our evaluation, we perform our proof-of-concept implementation of the proposed engine to demonstrate its feasibility and effectiveness, and we detect cyberthreats with an accuracy of 97.14%. This work is the first practical trial that detects cyberthreats from in-the-wild darknet traffic regardless of new types and variants in real time, and it quantitatively evaluates the result.

Dark-TRACER: Early Detection Framework for Malware Activity Based on Anomalous Spatiotemporal Patterns

“…This ground truth evaluation aimed to determine a hyperparameter set that minimized the number of false negatives, even if there were some false positives in each module, and evaluated the detection accuracy at that time. The conventional method, ChangeFinder, and the proposed modules, Dark-GLASSO and Dark-NMF, have already been tested and the results of those evaluations have been published [7], [8], whereas Dark-NTD was now evaluated for the first time using the same criteria. The following subsection describes the details of the dataset, the parameter tuning of Dark-NTD, and the comparison results from each module.…”

“…Proposed a method for real-time anomaly detection of spatiotemporal pattern synchronization using graphical lasso and graph density. 2020 [7] Quantitatively evaluated the accuracy of the method in paper [6] and discussed the results and limitations in depth.…”

“…To the best of our knowledge, there is no related work that focuses VOLUME 4, 2016 on synchronization in the same scope as the present study. As described in Table 9, prior papers [6] and [7] were published as prototypes of Dark-GLASSO, prior paper [8] was published as a prototype of Dark-NMF, and, finally, prior paper [9] was published as a prototype of Dark-NTD. However, as the previous methods have been considered independently, their relationship has remained unclear.…”

“…Next, as described in Section III-C, Dark-GLASSO performs random sampling preprocessing when the number of hosts 𝑁 ℎ exceeds 𝛾. In a previous paper [7], we reported that the output alerts were quite unstable when 𝛾 was lower than the expected average value of 𝑁 ℎ . Dark-NTD, as described in Section II-C, applies FSTD [20] to preprocess the tensor V to a low-rank approximation and preliminarily samples only the essential parts.…”

“…The NMF and NTD algorithms can decompose synchronous latent frequent patterns from data matrices or tensors into superpositions of multiple groups. We previously proposed the following different methods to estimate the synchronization in real time to automatically use the aforementioned algorithms and detect the source host space groups that show abnormal synchronization: Dark-GLASSO [6], [7], Dark-NMF [8], and Dark-NTD [9].…”