Improving Botnet Detection with Recurrent Neural Network and Transfer Learning

Abstract

Botnet detection is a critical step in stopping the spread of botnets and preventing malicious activities. However, reliable detection is still a challenging task, due to a wide variety of botnets involving ever-increasing types of devices and attack vectors. Recent approaches employing machine learning (ML) showed improved performance than earlier ones, but these MLbased approaches still have significant limitations. For example, most ML approaches can not incorporate sequential pattern analysis techniques key to detect some classes of botnets. Another common shortcoming of ML-based approaches is the need to retrain neural networks in order to detect the evolving botnets; however, the training process is time-consuming and requires significant efforts to label the training data. For fast-evolving botnets, it might take too long to create sufficient training samples before the botnets have changed again. To address these challenges, we propose a novel botnet detection method, built upon Recurrent Variational Autoencoder (RVAE) that effectively captures sequential characteristics of botnet activities. In the experiment, this semi-supervised learning method achieves better detection accuracy than similar learning methods, especially on hard to detect classes. Additionally, we devise a transfer learning framework to learn from a well-curated source data set and transfer the knowledge to a target problem domain not seen before. Tests show that the true-positive rate (TPR) with transfer learning is higher than the RVAE semi-supervised learning method trained using the target data set (91.8% vs. 68.3%).

APEX: Characterizing Attack Behaviors from Network Anomalies

“…As the first-level mining is performed without using the attack labels, this part of the training is unsupervised. Note that here we only mine flows originating from a single source IP address at a time; this is quite well-aligned with anomaly detection solutions that often predict the anomalous sources in a given time-slot since aggregation of flows at source level gives more information for better classification [2], [3].…”

“…Analyzing network traffic helps in detecting and identifying the threats and attacks faced by organizations. Past works have attempted to detect (often specific) attacks using supervised learning approaches that model the attack detection as a binary classification problem [1], [2], where labeled data of two classes of network traffic-normal and anomalous-are gathered and provided for training the models. However, network traffic (of, say, enterprises) tends to be noisy due to the large number of users, evolving landscape of applications and protocols (e.g., adoption of TLS 1.3 and HTTP/3), increasing use of new smart devices, etc.…”