Machine Learning With Variational AutoEncoder for Imbalanced Datasets in Intrusion Detection

Abstract

As a result of the explosion of security attacks and the complexity of modern networks, machine learning (ML) has recently become the favored approach for intrusion detection systems (IDS). However, the ML approach usually faces three challenges: massive attack variants, imbalanced data issues, and appropriate data segmentation. Improper handling of the issues will significantly degrade ML performance, e.g., resulting in high false-negative and low recall rates. Despite many efforts have done in the literature, detecting security attacks in a complicated network environment with imperfect data collection is still an open issue. This work proposes a machine learning framework with a combination of variational autoencoder and multilayer perceptron to deal with imbalanced datasets and detecting the explosion of attack variants on the Internet. The detection engine also includes an efficient range-based sequential search algorithm to address the segmentation challenge in data pre-processing from multiple sources (network packets, system/statistic logs) effectively. Our work is the first attempt to demonstrate the effect of using an appropriate combination of ML models for boosting IDS detection capability in a heterogeneous environment, where data collection imperfection is common. Experimental results on a public system log dataset (e.g., HDFS) show that our method gains approximately as much as 97% on F1 score and 98% on recall rate, a promising result compared to the same measurement of other solutions. Even better, we found that the proposed treatment of imbalanced datasets can improve up to 35% on the F1 score and 27% on recall rate. The testing results also indicate that our model can detect new attack variants. Code is available at https://github.com/tuonglinhhm/Hybrid-Learning-AutoEncoder-IDS.

Proactive ransomware prevention in pervasive IoMT via hybrid machine learning

“…Unsupervised learning has shown effectiveness in detecting ransomware in IoMT by identifying novel patterns and anomalies, as demonstrated by Zahoora et al [12] and Lin et al [13]. These studies utilized deep learning techniques like autoencoders and variational autoencoders (VAE) to analyze IoMT data, achieving high detection rates with low false positives.…”

Studying Imbalanced Learning for Anomaly-Based Intelligent IDS for Mission-Critical Internet of Things

“…Imbalanced datasets and improper data segmentation are the main contributors to a loss of the IDS detection accuracy. The authors of [34] proposed a machine learning framework in which they combined a variational autoencoder (VAE) and multilayer perceptron to simultaneously tackle the issues of imbalanced datasets (HDFS and TTP datasets) from multiple data sources and intrusions in complex, heterogeneous network environments. Using a hybrid learning approach, the authors used the variational autoencoder to address the imbalanced datasets issue in the training stage.…”

“…While the authors of [34] used a non-traditional approach to solve the data imbalance problem, the authors of [36] used traditional resampling methods (random undersampling, random oversampling, SMOTE, and ADA-SYN) to investigate the influences of these methods on the performance of artificial neural network (ANN) multi-class classifiers using benchmark cybersecurity datasets, including KDD99 and UNSW-NB18. Comparing the performance of the ANN multi-class classifiers using the resampling techniques, their study revealed that undersampling performed better than oversampling in terms of the training time, and oversampling performed better in terms of detecting minority data (abnormal examples).…”

Cyber Threats Detection in Smart Environments Using SDN-Enabled DNN-LSTM Hybrid Framework

“…However, along with numerous benefits, some prejudices have also adhered to these datasets. Lack of appropriate features for IoT, use of malevolent scripts for attack detection, and susceptibility to external cyber malfunctions are some of such enmities [54]. We have adopted the CICIDS2018 dataset which is remarkably known for its spacious range of features towards IoT communications [55,56].…”