Highly accurate and efficient two phase-intrusion detection system (TP-IDS) using distributed processing of HADOOP and machine learning techniques

Abstract

Network security and data security are the biggest concerns now a days. Every organization decides their future business process based on the past and day to day transactional data. This data may consist of consumer’s confidential data, which needs to be kept secure. Also, the network connections when established with the external communication devices or entities, a care should be taken to authenticate these and block the unwanted access. This consists of identification of the malicious connection nodes and identification of normal connection nodes. For that, we use a continuous monitoring of the network input traffic to recognize the malicious connection request called as intrusion and this type of monitoring system is called as an Intrusion detection system (IDS). IDS helps us to protect our network and data from insecure and malicious network connections. Many such systems exists in the real time scenario, but they have critical issues of performance like accuracy and efficiency. These issues are addressed as a part of this research work of IDS using machine learning techniques and HDFS. The TP-IDS is designed in two phases for increasing accuracy. In phase I of TP-IDS, Support Vector Machine (SVM) and k Nearest Neighbor (kNN) are used. In phase II of TP-IDS, Decision Tree (DT) and Naïve Bayes (NB) are used, where phase II is the validation phase of the system for increasing accuracy. Also, both the phases are having Hadoop distributed file system underlying data storage and processing architecture, which allows parallel processing to increase the speed of the system and hence achieve the efficiency in TP-IDS.

Research on the Application of Random Forest-based Feature Selection Algorithm in Data Mining Experiments

“…Posttraining, the model employs an auto-encoder for prediction, significantly reducing detection time and enhancing accuracy. Experimental findings suggest that AE-IDS outperform conventional ML-based IDS, offering more accessible training, better adaptability, and higher detection accuracy in [46][47][48][49][50]. The research in [51][52][53][54][55] employs an RF algorithm for county-scale cotton mapping, using spectral, vegetation, and www.ijacsa.thesai.org texture features.…”

Accuracy Based Fault Tolerant Two Phase - Intrusion Detection System (TP-IDS) Using Machine Learning and HDFS

“…Decision tree are constructed with decision nodes as internal nodes and the class labels as the leaf nodes of the tree. Each internal node is defined for the decision for classification path, based on the values of different features used for classification as input variables [19]. There are number of algorithms used for construction of the decision tree such as ID3, C4.5, VFDT, etc.…”

Implement an Intrusion Detection System Utilizing Machine Learning and Principal Component Analysis

“…Previous methods such as data encryption, firewalls, and user authentication are insufficient to prevent Internet attacks, so cyber solutions that work to avoid anomalies and attacks [2] came to protect data from attack. Intrusion detection systems monitor network traffic and prevent intrusion.…”