AUTO-CDD: automatic cleaning dirty data using machine learning techniques

Jesmeen, M. Z. H.; Hossen, Abid; Hossen, J.; Raja, J. Emerson; Bhuvaneswari, T.; Sayeed, Shohel

doi:10.12928/telkomnika.v17i4.12780

Cited by 6 publications

(3 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The evaluation on the basis of Accuracy (ACC) value is executed. Accuracy measures the degree to which the instances correctly classified by machine learning algorithm and can be computed using a confusion matrix with (5) as follows [23]:…”

Section: Results and Analysismentioning

confidence: 99%

Developed third iterative dichotomizer based on feature decisive values for educational data mining

Ahmed

Alhamdani

Croock

2020

IJEECS

View full text Add to dashboard Cite

Recently, the decision trees have been adopted among the preeminent utilized classification models. They acquire their fame from their efficiency in predictive analytics, easy to interpret and implicitly perform feature selection. This latter perspective is one of essential significance in Educational Data Mining (EDM), in which selecting the most relevant features has a major impact on classification accuracy enhancement. The main contribution is to build a new multi-objective decision tree, which can be used for feature selection and classification. The proposed Decisive Decision Tree (DDT) is introduced and constructed based on a decisive feature value as a feature weight related to the target class label. The traditional Iterative Dichotomizer 3 (ID3) algorithm and the proposed DDT are compared using three datasets in terms of some ID3 issues, including logarithmic calculation complexity and multi-values featuresselection. The results indicated that the proposed DDT outperforms the ID3 in the developing time. The accuracy of the classification is improved on the basis of 10-fold cross-validation for all datasets with the highest accuracy achieved by the proposed method is 92% for the student.por dataset and holdout validation for two datasets, i.e. Iraqi and Student-Math. The experiment also shows that the proposed DDT tends to select attributes that are important rather than multi-value.

show abstract

Section: Results and Analysismentioning

confidence: 99%

Developed third iterative dichotomizer based on feature decisive values for educational data mining

Ahmed

Alhamdani

Croock

2020

IJEECS

View full text Add to dashboard Cite

show abstract

“…Jesmeen et al 23 presented a comparison between currently used algorithms and their proposed tool, Auto-CDD, to handle missing values. The developed system improved overall data processing and guaranteed to overcome processing unwanted outcomes in data analysis.…”

Section: Related Workmentioning

confidence: 99%

Smartic: A smart tool for Big Data analytics and IoT

2022

View full text Add to dashboard Cite

The Internet of Things (IoT) is leading the physical and digital world of technology to converge. Real-time and massive scale connections produce a large amount of versatile data, where Big Data comes into the picture. Big Data refers to large, diverse sets of information with dimensions that go beyond the capabilities of widely used database management systems, or standard data processing software tools to manage within a given limit. Almost every big dataset is dirty and may contain missing data, mistyping, inaccuracies, and many more issues that impact Big Data analytics performances. One of the biggest challenges in Big Data analytics is to discover and repair dirty data; failure to do this can lead to inaccurate analytics results and unpredictable conclusions. We experimented with different missing value imputation techniques and compared machine learning (ML) model performances with different imputation methods. We propose a hybrid model for missing value imputation combining ML and sample-based statistical techniques. Furthermore, we continued with the best missing value inputted dataset, chosen based on ML model performance for feature engineering and hyperparameter tuning. We used k-means clustering and principal component analysis. Accuracy, the evaluated outcome, improved dramatically and proved that the XGBoost model gives very high accuracy at around 0.125 root mean squared logarithmic error (RMSLE). To overcome overfitting, we used K-fold cross-validation.

show abstract

“…It was proved by different resources about the loss of billions of dollars due to poor DQ [11], [12]. Low-level DQ can cause due to wrong or missing data and is very essential to handle this type of dataset [13]. It may lead to incorrect or misleading decisions, predictions, or instructions.…”

Section: Introductionmentioning

confidence: 99%

Anomalies detection for smart-home energy forecasting using moving average

Jesmeen

Murthy

Hossen

et al. 2022

IJECE

View full text Add to dashboard Cite

In the past few years, the increase in the relation between the physical and digital world over the internet was witnessed. Even though the applications can enhance smart home systems, it is still early stages and challenges in the field of internet of things (IoT). An extreme level of data quality (DQ) system management is essential to produce a meaningful vision. However, in most home energy management system has no straightforward process of removing abnormal data. Hence, the research aims to propose and validate the model of anomaly detection for power consumption in real-time. The moving average (MA) approach identifies and removes abnormal energy consumption data. The results obtained from the forecasting time series auto regressive integrated moving average (ARIMA) model demonstrated that the proposed heuristics effectively enhanced energy usage forecasting. The selection of optimum parameter values for the MA was comprehended for time-series forecasting error minimization by comparing mean squared error (MSE). These outcomes proved the effectiveness of the existing technique and precision of choice of the appropriate. Therefore, the method can effectively route the cleaned sequence data streams in a real-time environment, which is valuable for spotting the anomalies and eliminating for enhancing energy usage time series.

show abstract

AUTO-CDD: automatic cleaning dirty data using machine learning techniques

Cited by 6 publications

References 18 publications

Developed third iterative dichotomizer based on feature decisive values for educational data mining

Developed third iterative dichotomizer based on feature decisive values for educational data mining

Smartic: A smart tool for Big Data analytics and IoT

Anomalies detection for smart-home energy forecasting using moving average

Contact Info

Product

Resources

About