Spatio-temporal factorization of log data for understanding network events

Kimura, Toshikiro; Ishibashi, Keisuke; Mori, Tatsuya; Sawada, Hiroshi; Toyono, Tsuyoshi; Nishimatsu, Ken; Watanabe, Akio; Shimoda, Akihiro; Shiomoto, Kohei

doi:10.1109/infocom.2014.6847986

Cited by 68 publications

(30 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are methods of giving new insights for operators with knowledge mining from log data [10], [11]. Kobayashi et al [7] proposed a time series causal inference method in network logs.…”

Section: Related Workmentioning

confidence: 99%

Latent Variable Based Anomaly Detection in Network System Logs

Otomo

Kobayashi

Fukuda

et al. 2019

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for large-scale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 12 months of system logs obtained from a nationwide academic network in Japan. Through comparisons with Kleinberg's univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method achieves 14.5% higher recall and 3% higher precision than PCA. A case study shows detected anomalies are effective information for troubleshooting of network system faults.

show abstract

“…There are methods of giving new insights for operators with knowledge mining from log data [10], [11]. Kobayashi et al [7] proposed a time series causal inference method in network logs.…”

Section: Related Workmentioning

confidence: 99%

Latent Variable Based Anomaly Detection in Network System Logs

Otomo

Kobayashi

Fukuda

et al. 2019

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

show abstract

“…We acknowledge that logs also contain text parameters and our approach can also deal with these type of messages. STE [15] takes another method using the following features of log messages: parameters appear less frequently than template words. The approach gives a score to each word in the log and decides if the word belongs to the templates by using DBSCAN algorithm.…”

Section: Related Workmentioning

confidence: 99%

“…Besides, many log correlation analysis approaches utilize parameter-based causality analysis [22,23]. This can also cause significant false relations because they do not uncover the real logical or semantic relations among logs or events [15]. Furthermore, it is difficult to define or identify the events in log entries [7].…”

Section: Related Workmentioning

confidence: 99%

Anomalies Detection and Proactive Defence of Routers Based on Multiple Information Learning

Teng

Shen

et al. 2019

Entropy

View full text Add to dashboard Cite

Routers are of great importance in the network that forward the data among the communication devices. If an attack attempts to intercept the information or make the network paralyzed, it can launch an attack towards the router and realize the suspicious goal. Therefore, protecting router security has great importance. However, router systems are notoriously difficult to understand or diagnose for their inaccessibility and heterogeneity. A common way of gaining access to the router system and detecting the anomaly behaviors is to inspect the router syslogs or monitor the packets of information flowing to the routers. These approaches just diagnose the routers from one aspect but do not correlate multiple logs. In this paper, we propose an approach to detect the anomalies and faults of the routers with multiple information learning. First, we do the offline learning to transform the benign or corrupted user actions into the syslogs. Then, we construct the log correlation among different events. During the detection phase, we calculate the distance between the event and the cluster to decide if it is an anomalous event and we use the attack chain to predict the potential threat. We applied our approach in a university network which contains Huawei, Cisco and Dlink routers for three months. We aligned our experiment with former work as a baseline for comparison. Our approach obtained 89.6% accuracy in detecting the attacks, which is 5.1% higher than the former work. The results show that our approach performs in limited time as well as memory usages and has high detection and low false positives.

show abstract

“…Kimura et al adopted a supervised machine learning technique to associate network failures with the network log data that appeared in the past using network trouble ticket data and developed an online template extraction method and a future extraction method that characterizes the abnormality of logs based on the generation patterns of logs. Kimura et al proposed a modeling and event extraction method on network log data using a tensor factorization approach. Although the above researches are able to automatically extract features from the log files, but their log format is predefined, so that it is relatively easy to extract features and template.…”

Section: Related Workmentioning

confidence: 99%

A feature extraction method for predictive maintenance with time‐lagged correlation–based curve‐registration model

Zhang

Liu

et al. 2018

Int J Network Mgmt

View full text Add to dashboard Cite

With the prevalent development and use of predictive maintenance models for Internet‐of‐Things scenarios, the deep learning technology is gaining momentum. Feature extraction helps to increase efficiency in training the deep‐learning–based predictive maintenance model. However, there are common situations of time‐lagged correlations among industrial sensor data, resulting in reduction the effect of feature extraction. In this paper, we propose a feature extraction method for multisensors data with time‐lagged correlation. A curve‐registration method of correlation maximization algorithm is used to solve the problem of time‐lagged correlation for multi sensors. Then we apply a recurrent neural network, namely, long short‐term memory to develop a lightweight predictive maintenance model with the help of proposed feature extraction method. The effectiveness of the proposed feature extraction approach is demonstrated by examining real cases in a power plant. The experimental results indicate that our method can (1) effectively improve the accuracy of prediction and (2) improve the performance of the prediction model.

show abstract

Spatio-temporal factorization of log data for understanding network events

Cited by 68 publications

References 16 publications

Latent Variable Based Anomaly Detection in Network System Logs

Latent Variable Based Anomaly Detection in Network System Logs

Anomalies Detection and Proactive Defence of Routers Based on Multiple Information Learning

A feature extraction method for predictive maintenance with time‐lagged correlation–based curve‐registration model

Contact Info

Product

Resources

About