Overcoming the Challenges of Data Lack, Leakage, and Dimensionality in Intrusion Detection Systems: A Comprehensive Review

Abstract

The Internet of Things (IoT) and cloud computing are rapidly gaining momentum as decentralized internet-based technologies and have led to an increase in information in nearly every technical and commercial industry. However, ensuring the security of IoT systems is a pressing issue due to the complexities involved in connected and shared environments. Networks are guarded by Intrusion Detection Systems (IDS) against various cyber threats such as malware, viruses, and unauthorized access. IDS have recently adopted Machine Learning (ML) and Deep Learning (DL) techniques to identify and classify security risks. However, the effective utilization of these technologies depends on the availability, quality, and characteristics of the data used to train models. Moreover, data lack, data leak, and dimensionality (DLLD) are common problems in data science and ML. This paper surveys existing research and suggests solutions for overcoming DLLD-related issues to improve IDS model.

Implications of Data Leakage in Machine Learning Preprocessing: A Multi-Domain Investigation

“…Temporal leakage is a frequent issue in time-series data, occurring when models are trained using future data. Lastly, data preprocessing leakage may happen during normalization and encoding steps if they involve information from the full dataset [10]- [12].…”