Deleting or keeping outliers for classifier training?

Tallón‐Ballesteros, Antonio J.; Riquelme, José C.

doi:10.1109/nabic.2014.6921892

Cited by 19 publications

(18 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The findings indicated that omitting outliers from the training data considerably refined the kNN classifier. In a general case study, Tallon-Ballesteros and Riquelme evaluated the outlier effect in classification problems [48]. The study proposed a statistical outlier detection method to determine the outliers based on inter-quartile range (lQR) by classes.…”

Section: Outlier Detection Methodsmentioning

confidence: 99%

Hybrid Prediction Model for Type 2 Diabetes and Hypertension Using DBSCAN-Based Outlier Detection, Synthetic Minority Over Sampling Technique (SMOTE), and Random Forest

et al. 2018

View full text Add to dashboard Cite

As the risk of diseases diabetes and hypertension increases, machine learning algorithms are being utilized to improve early stage diagnosis. This study proposes a Hybrid Prediction Model (HPM), which can provide early prediction of type 2 diabetes (T2D) and hypertension based on input risk-factors from individuals. The proposed HPM consists of Density-based Spatial Clustering of Applications with Noise (DBSCAN)-based outlier detection to remove the outlier data, Synthetic Minority Over-Sampling Technique (SMOTE) to balance the distribution of class, and Random Forest (RF) to classify the diseases. Three benchmark datasets were utilized to predict the risk of diabetes and hypertension at the initial stage. The result showed that by integrating DBSCAN-based outlier detection, SMOTE, and RF, diabetes and hypertension could be successfully predicted. The proposed HPM provided the best performance result as compared to other models for predicting diabetes as well as hypertension. Furthermore, our study has demonstrated that the proposed HPM can be applied in real cases in the IoT-based Health-care Monitoring System, so that the input risk-factors from end-user android application can be stored and analyzed in a secure remote server. The prediction result from the proposed HPM can be accessed by users through an Android application; thus, it is expected to provide an effective way to find the risk of diabetes and hypertension at the initial stage.

show abstract

Section: Outlier Detection Methodsmentioning

confidence: 99%

Hybrid Prediction Model for Type 2 Diabetes and Hypertension Using DBSCAN-Based Outlier Detection, Synthetic Minority Over Sampling Technique (SMOTE), and Random Forest

et al. 2018

View full text Add to dashboard Cite

show abstract

“…Previous studies showed that removing the outlier can improve the classification accuracy. Tallón-Ballesteros and Riquelme utilized outlier detection for a classification model [ 50 ]. The authors proposed a statistical outlier detection method based on the interquartile range (lQR) with classes.…”

Section: Literature Reviewmentioning

confidence: 99%

“…However, machine learning algorithms encounter problems with outlier data, which can reduce the accuracy of the classification model. Outlier detection can be applied to identify and remove outliers; thus, improving the performance of classification models [ 50 , 51 ]. One of the techniques used for outlier detection is Density-Based Spatial Clustering of Applications with Noise (DBSCAN) [ 52 ].…”

Section: Introductionmentioning

confidence: 99%

Performance Analysis of IoT-Based Sensor, Big Data Processing, and Machine Learning Model for Real-Time Monitoring System in Automotive Manufacturing

Syafrudin

Alfian

Fitriyani

et al. 2018

Sensors

278

103

View full text Add to dashboard Cite

With the increase in the amount of data captured during the manufacturing process, monitoring systems are becoming important factors in decision making for management. Current technologies such as Internet of Things (IoT)-based sensors can be considered a solution to provide efficient monitoring of the manufacturing process. In this study, a real-time monitoring system that utilizes IoT-based sensors, big data processing, and a hybrid prediction model is proposed. Firstly, an IoT-based sensor that collects temperature, humidity, accelerometer, and gyroscope data was developed. The characteristics of IoT-generated sensor data from the manufacturing process are: real-time, large amounts, and unstructured type. The proposed big data processing platform utilizes Apache Kafka as a message queue, Apache Storm as a real-time processing engine and MongoDB to store the sensor data from the manufacturing process. Secondly, for the proposed hybrid prediction model, Density-Based Spatial Clustering of Applications with Noise (DBSCAN)-based outlier detection and Random Forest classification were used to remove outlier sensor data and provide fault detection during the manufacturing process, respectively. The proposed model was evaluated and tested at an automotive manufacturing assembly line in Korea. The results showed that IoT-based sensors and the proposed big data processing system are sufficiently efficient to monitor the manufacturing process. Furthermore, the proposed hybrid prediction model has better fault prediction accuracy than other models given the sensor data as input. The proposed system is expected to support management by improving decision-making and will help prevent unexpected losses caused by faults during the manufacturing process.

show abstract

“…A statistical outlier detection method based on the partial removal of outliers according the inter-quartile range of all the instances with the same class label was introduced in [22] with the name OUTLIERoutP. The framework can be reviewed in Figure 2 of the aforementioned work.…”

Section: Proposalmentioning

confidence: 99%

“…According to the literature, the researches tackle data cleansing or feature selection in an isolated way. Table 1 describes the data sets utilised together with the outlier level according to the taxonomy proposed in [22]. Most of them are publicly available in the UCI (University of California at Irvine) repository [4].…”

Section: Proposalmentioning

confidence: 99%

Data Cleansing Meets Feature Selection: A Supervised Machine Learning Approach

Tallón‐Ballesteros

Riquelme

2015

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. This paper presents a novel procedure to apply in a sequential way two data preparation techniques from a different nature such as data cleansing and feature selection. For the former we have experienced with a partial removal of outliers via inter-quartile range whereas for the latter we have chosen relevant attributes with two widespread feature subset selectors like CFS (Correlation-based Feature Selection) and CNS (Consistency-based Feature Selection), which are founded on correlation and consistency measures, respectively. Empirical results on seven difficult binary and multi-class data sets, that is, with a test error rate of at least a 10%, according to accuracy, with C4.5 or 1-nearest neighbour classifiers without any kind of prior data pre-processing are outlined. Non-parametric statistical tests assert that the meeting of the aforementioned two data preparation strategies using a correlation measure for feature selection with C4.5 algorithm is significant better, measured with roc measure, than the single application of the data cleansing approach. Last but not least, a weak and not very powerful learner like PART achieved promising results with the new proposal based on a consistency measure and is able to compete with the best configuration of C4.5. To sum up, bearing in mind the new approach, for roc measure PART classifier with a consistency metric behaves slightly better than C4.5 and a correlation measure.

show abstract

Deleting or keeping outliers for classifier training?

Cited by 19 publications

References 11 publications

Hybrid Prediction Model for Type 2 Diabetes and Hypertension Using DBSCAN-Based Outlier Detection, Synthetic Minority Over Sampling Technique (SMOTE), and Random Forest

Hybrid Prediction Model for Type 2 Diabetes and Hypertension Using DBSCAN-Based Outlier Detection, Synthetic Minority Over Sampling Technique (SMOTE), and Random Forest

Performance Analysis of IoT-Based Sensor, Big Data Processing, and Machine Learning Model for Real-Time Monitoring System in Automotive Manufacturing

Data Cleansing Meets Feature Selection: A Supervised Machine Learning Approach

Contact Info

Product

Resources

About