Hybrid Pre-processing Technique for Handling Imbalanced Data and Detecting Outliers for KNN Classifier

Nair, Preeti; Kashyap, Indu

doi:10.1109/comitcon.2019.8862250

Cited by 32 publications

(18 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Moreover, we selected the optimal sample size for MC simulation by measuring the percentage of variance of the PCA components trained on the MC pre-processed dataset explained by the PCA components trained on the IQR pre-processed dataset in terms of R 2 due to many considerations in the literature which report pre-processing techniques for similar anomaly detection scenarios based on the IQR method [24]. This analysis led to an optimal value of three samples to be considered for the median computation.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Anomaly Detection in Photovoltaic Production Factories via Monte Carlo Pre-Processed Principal Component Analysis

Arena

Corsini

Ferulano³

et al. 2021

Energies

View full text Add to dashboard Cite

This paper investigates a use case of robust anomaly detection applied to the scenario of a photovoltaic production factory—namely, Enel Green Power’s 3SUN solar cell production plant in Catania, Italy—by considering a Monte Carlo based pre-processing technique as a valid alternative to other typically used methods. In particular, the proposed method exhibits the following advantages: (i) Outlier replacement, by contrast with traditional methods which are limited to outlier detection only, and (ii) the preservation of temporal locality with respect to the training dataset. After pre-processing, the authors trained an anomaly detection model based on principal component analysis and defined a suitable key performance indicator for each sensor in the production line based on the model errors. In this way, by running the algorithm on unseen data streams, it is possible to isolate anomalous conditions by monitoring the above-mentioned indicators and virtually trigger an alarm when exceeding a reference threshold. The proposed approach was tested on both standard operating conditions and an anomalous scenario. With respect to the considered use case, it successfully anticipated a fault in the equipment with an advance of almost two weeks, but also demonstrated its robustness to false alarms during normal conditions.

show abstract

Section: Discussionmentioning

confidence: 99%

“…The related literature also reports pre-processing techniques for similar anomaly detection scenarios based on the IQR method (e.g., [24]), which, however, offers only the property of outlier removal and not the additional benefit of outlier replacement that is consequential to applying MC simulation, as further discussed in Section 3.…”

Section: Related Workmentioning

confidence: 99%

Anomaly Detection in Photovoltaic Production Factories via Monte Carlo Pre-Processed Principal Component Analysis

Arena

Corsini

Ferulano³

et al. 2021

Energies

View full text Add to dashboard Cite

show abstract

“…In paper [3] the authors have explored ways to increase the efficiency of KNN algorithm so that it can give out better results. The evolutionary Genetic Algorithm is used to select the finest parameters of the nonlinear functions that are suitable for each feature, and the results are better comparatively and on similar lines in paper [4], Preeti Nair and Indu Kashyap have made implored that by introducing resample technique and Inter quartile range technique (IQR) in the pre-processing steps the data fed to classifiers are normalized which gives out better working of the algorithm.…”

Section: Related Workmentioning

confidence: 92%

Fake News Detection from Online media using Machine learning Classifiers

Pandey¹,

Prabhakaran²,

Reddy³

et al. 2022

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

With the advancement in technology, the consumption of news has shifted from Print media to social media. The convenience and accessibility are major factors that have contributed to this shift in consumption of the news. However, this change has bought upon a new challenge in the form of “Fake news” being spread with not much supervision available on the net. In this paper, this challenge has been addressed through a Machine learning concept. The algorithms such as K-Nearest Neighbor, Support Vector Machine, Decision Tree, Naïve Bayes and Logistic regression Classifiers to identify the fake news from real ones in a given dataset and also have increased the efficiency of these algorithms by pre-processing the data to handle the imbalanced data more appropriately. Additionally, comparison of the working of these classifiers is presented along with the results. The model proposed has achieved an accuracy of 89.98% for KNN, 90.46% for Logistic Regression, 86.89% for Naïve Bayes, 73.33% for Decision Tree and 89.33% for SVM in our experiment.

show abstract

“…Dataset yang memiliki missing values, noisy data dan imbalanced dataset dapat mempengaruhi hasil klasifikasi dan akurasi [3]. Missing values ditemukan jika dalam suatu dataset ada data yang hilang pada suatu atributnya [4]. Noisy data ditemukan jika dalam suatu dataset ada data yang di luar range atau tidak berhubungan [5].…”

Section: Pendahuluanunclassified

“…Normalisasi atribut adalah proses konversi data untuk suatu atribut dari numerik menjadi skala secara umum dan sebaliknya, tanpa menganggu variasi datanya [8]. Imbalanced dataset terjadi jika dalam suatu dataset perbandingan dari jumlah label yang tidak seimbang atau terlalu jauh [4]. Dalam metode klasifikasi, label dijadikan acuan untuk mengukur performa dalam model yang dihasilkan [9], sehingga jika dalam suatu dataset salah satu label atau class-nya mempunyai jumlah yang lebih besar secara signifikan dibanding label lainnya, maka kemungkinan pengujian ata hasil klasifikasi dengan data uji akan mempunyai kecederungan lebih dekat pada label yang mempunyai jumlah yang lebih besar pada proses trainingnya.…”

Section: Pendahuluanunclassified

Performance Analysis of Random Forest Using Attribute Normalization

Nugroho¹,

Abdullah

2022

SISTEMASI

View full text Add to dashboard Cite

Data mining dapat memproses data masa lalu menjadi pola untuk membantu aktivitas manusia pada masa berikutnya. Dalam data mining terbagi menjadi beberapa metode, yaitu klasifikasi, klustering, asosisi dan peramalan. Dalam penelitian ini, menggunakan metode klasifikasi untuk menentukan pola dari suatu dataset, sehingga dapat digunakan untuk prediksi keputusan dengan data yang baru. Dataset untuk metode klasifikasi harus mempunyai label atau class. Dataset yang mempunyai label yang jumlahnya tidak seimbang (imbalanced dataset) dapat mempengaruhi bentuk model dan hasil prediksi untuk data yang baru. Untuk mengatasi masalah tersebut, dalam penelitian ini menggunakan ensemble method dan pre-processing. Salah satu algoritma dalam ensemble learning method adalah random forest dan pre-processing yang digunakan adalah normalisasi atribut dengan mengubah data nominal menjadi numerik. Random Forest merupakan pengembangan dari decision tree yang menghasilkan pola berbentuk pohon, dimana pola ini dapat menunjukkan alur dari proses klasifikasi. Random forest akan digunakan untuk proses pembelajaran pada data setelah proses normalisasi atribut dilakukan. Tujuan penelitian ini adalah menerapkan proses normalisasi atribut dan menggunakan algoritma random forest untuk mengatasi imbalanced dataset dan mengukur akurasi. Penelitian ini menggunakan dataset publik dari UCI Repository, yaitu car evaluation. Akurasi yang dihasilkan dengan metode ini ± 99 % dengan 90 % data training dan 10 % data testing dan ± 95,95% dengan delapan k-folds cross-validation dan jumlah pohon 100 pohon.

show abstract

Hybrid Pre-processing Technique for Handling Imbalanced Data and Detecting Outliers for KNN Classifier

Cited by 32 publications

References 6 publications

Anomaly Detection in Photovoltaic Production Factories via Monte Carlo Pre-Processed Principal Component Analysis

Anomaly Detection in Photovoltaic Production Factories via Monte Carlo Pre-Processed Principal Component Analysis

Fake News Detection from Online media using Machine learning Classifiers

Performance Analysis of Random Forest Using Attribute Normalization

Contact Info

Product

Resources

About