Encoding NetFlows for State-Machine Learning

Abstract

NetFlow data is a well-known network log format used by many network analysts and researchers. The advantages of using this format compared to pcap are that it contains fewer data, is less privacy intrusive, and is easier to collect and process. However, having less data does mean that this format might not be able to capture important network behaviour as all information is summarised into statistics. Much research aims to overcome this disadvantage through the use of machine learning, for instance, to detect attacks within a network. Many approaches can be used to pre-process the NetFlow data before it is used to train the machine learning algorithms. However, many of these approaches simply apply existing methods to the data, not considering the specific properties of network data. We argue that for data originating from software systems, such as NetFlow or software logs, similarities in frequency and contexts of feature values are more important than similarities in the value itself. In this work, we, therefore, propose an encoding algorithm that directly takes the frequency and the context of the feature values into account when the data is being processed. Different types of network behaviours can be clustered using this encoding, thus aiding the process of detecting anomalies within the network. From windows of these clusters obtained from monitoring a clean system, we learn state machine behavioural models for anomaly detection. These models are very well-suited to modelling the cyclic and repetitive patterns present in NetFlow data. We evaluate our encoding on a new dataset that we created for detecting problems in Kubernetes clusters and on two wellknown public NetFlow datasets. The obtained performance results of the state machine models are comparable to existing works that use many more features and require both clean and infected data as training input.

SoK: Explainable Machine Learning for Computer Security Applications

“…Source port is problematic because it typically gets arbitrarily assigned by the operating system, and as such should not be indicative of malicious behaviour. However, the CTU-13 dataset uses only a small subset of VMrelated port numbers [29], which inadvertently becomes indicative of malicious behaviour. This is a common shortcoming of lab-collected datasets [16].…”

Anomaly network intrusion detection system based on NetFlow using machine/deep learning

“…Compared to pcap format, NetFlow data contains less data, making it easier to collect and process. Additionally, NetFlow is less intrusive to privacy, further enhancing its appeal as a preferred network log format (Cao et al, 2022). Rather than focusing on individual packets, flow monitoring analyzes the flow of traffic, making it a more scalable approach to traffic analysis.…”

“…Every flow in NetFlow contains network statistics representing a connection between two hosts. These statistics can be utilized to compute performance metrics and to identify any unusual or abnormal network behavior (Cao et al, 2022).…”