LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things

Abstract

Log anomaly detection is an efficient method to manage modern large-scale Internet of Things (IoT) systems. More and more works start to apply natural language processing (NLP) methods, and in particular word2vec, in the log feature extraction. Word2vec can extract the relevance between words and vectorize the words. However, the computing cost of training word2vec is high. Anomalies in logs are dependent on not only an individual log message but also on the log message sequence. Therefore, the vector of words from word2vec can not be used directly, which needs to be transformed into the vector of log events and further transformed into the vector of log sequences. To reduce computational cost and avoid multiple transformations, in this paper, we propose an offline feature extraction model, named LogEvent2vec, which takes the log event as input of word2vec to extract the relevance between log events and vectorize log events directly. LogEvent2vec can work with any coordinate transformation methods and anomaly detection models. After getting the log event vector, we transform log event vector to log sequence vector by bary or tf-idf and three kinds of supervised models (Random Forests, Naive Bayes, and Neural Networks) are trained to detect the anomalies. We have conducted extensive experiments on a real public log dataset from BlueGene/L (BGL). The experimental results demonstrate that LogEvent2vec can significantly reduce computational time by 30 times and improve accuracy, comparing with word2vec. LogEvent2vec with bary and Random Forest can achieve the best F1-score and LogEvent2vec with tf-idf and Naive Bayes needs the least computational time.

A Novel Dynamic Three-Level Tracking Controller for Mobile Robots Considering Actuators and Power Stage Subsystems: Experimental Assessment

“…Finally, future research will be focused on solving the obstacle avoidance and the path following control tasks by using the approach presented in this paper. On the other hand, and with the aim of possibly generating a new direction for this research topic, it would be interesting to implement some kind of learning technology such as machine learning [ 45 ] or Internet of things (IoT) [ 46 , 47 ]. Also, using wireless network sensors (WSNs) [ 48 ] could be another interesting path for highlighting even more the research proposed in this paper.…”

LogUAD: Log Unsupervised Anomaly Detection Based on Word2Vec

“…LSTM model needs to be trained with a large amount of normal data, but the original dataset cannot be all normal which may contain some abnormal log. These works can not cope with the evolution of log statements, especially in the case of new log templates [17], which greatly limits their applicability in practice. Log instability is common, but there are few works on it.…”

Correlating Time Series Signals and Event Logs in Embedded Systems

“…Within time series analysis, we distinguish four research goals: (i) TS decomposition [3,8], which it involves deriving trend, season, noise and other specified components; (ii) TS classification [9][10][11], based on comparing time series with each other and finding similarities using diverse metrics; (iii) deriving characteristic and anomalous features [12][13][14]; and (iv) predicting future behaviour [3,4,15] by finding data patterns for the near future based on the previous/historical ones. They are considered as general or applicationtargeted problems.…”