Feature Extraction and Feature Selection : Reducing Data Complexity with Apache Spark

Sisiaridis, Dimitrios; Markowitch, Olivier

doi:10.5121/ijnsa.2017.9604

Cited by 4 publications

(2 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A comprehensive study was carried out based on the efficiency of several classifier models using Spark MLlib [33], [34]. A variety of evaluation parameters have been analyzed to determine the best one.…”

Section: Results and Analysismentioning

confidence: 99%

Spark-MLlib intrusion detection mechanism using machine learning models

Sarwath,

Gulmeher,

Sultana

2024

IJEECS

View full text Add to dashboard Cite

<p align="justify"><span>Typically, a single method is employed in machine learning (ML) based intrusion detection to identify intrusion information. However, this approach lacks flexibility, has a low detection rate, and struggles to handle high-dimensional data. Consequently, it is not efficient in addressing these challenges. This study proposes a new intrusion detection architecture that utilizes Spark and ensures resilient data dissemination across the platform to improve its effectiveness. It consists of preprocessing module, a label encoder module, a feature extraction module, a classification module and a database module. The preprocessing module compresses information by utilizing the module for label encoding. This generates a lower-dimensional reconstruction and classification characteristic. The database module has the capability to store the compressed characteristics of all traffic. This enables the classifier to be tested and then returns these features back into the original traffic, facilitating retraining. In order to evaluate the efficacy of the framework, simulations were conducted using the CICIDS 2017 dataset to accurately replicate the network traffic. Based on the test findings, the accuracy of both multiclass and binary classification surpasses that of earlier studies. High precision was achieved for the traffic that was restored. The possible application of the proposed architecture for edge/fog networks is discussed in the conclusion.</span></p>

show abstract

Section: Results and Analysismentioning

confidence: 99%

Spark-MLlib intrusion detection mechanism using machine learning models

Sarwath,

Gulmeher,

Sultana

2024

IJEECS

View full text Add to dashboard Cite

show abstract

“…However, time series clustering is a complex problem due to its high data dimensionality, which may cause problems such as highly biased estimates and decreases clustering performance [6]. Therefore, dimensionality reduction techniques play an important role in this field [8] by transforming a high-dimensional data representation into a lower low-dimensional data representation [9].…”

Section: Introductionmentioning

confidence: 99%

KPIs-Based Clustering and Visualization of HPC Jobs: A Feature Reduction Approach

2021

View full text Add to dashboard Cite

High-Performance Computing (HPC) systems need to be constantly monitored to ensure their stability. The monitoring systems collect a tremendous amount of data about different parameters or Key Performance Indicators (KPIs), such as resource usage, IO waiting time, etc. A proper analysis of this data, usually stored as time series, can provide insight in choosing the right management strategies as well as the early detection of issues. In this paper, we introduce a methodology to cluster HPC jobs according to their KPI indicators. Our approach reduces the inherent high dimensionality of the collected data by applying two techniques to the time series: literature-based and variance-based feature extraction. We also define a procedure to visualize the obtained clusters by combining the two previous approaches and the Principal Component Analysis (PCA). Finally, we have validated our contributions on a real data set to conclude that those KPIs related to CPU usage provide the best cohesion and separation for clustering analysis and the good results of our visualization methodology. INDEX TERMS Clustering, feature extraction, high-performance computing, time series analysis.

show abstract