A Review on Data Cleansing Methods for Big Data

Ridzuan, Fakhitah; Zainon, Wan Mohd Nazmee Wan

doi:10.1016/j.procs.2019.11.177

Cited by 147 publications

(83 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For tackling issue (b), namely data heterogeneity, it is crucial to identify appropriate data cleaning techniques; see, e.g., Rahm and Hai Do (2000) and Ridzuan et al (2019) for overviews of traditional techniques and extensions for handling big data, respectively. In general, the elimination of noise (of which redundancy is one of the most prominent aspects) is recognized as fundamental for obtaining high-quality datasets and, at the same time, is an interactive, error prone and time consuming activity.…”

Section: Discussionmentioning

confidence: 99%

Merging Datasets of CyberSecurity Incidents for Fun and Insight

et al. 2021

View full text Add to dashboard Cite

Providing an adequate assessment of their cyber-security posture requires companies and organisations to collect information about threats from a wide range of sources. One of such sources is history, intended as the knowledge about past cyber-security incidents, their size, type of attacks, industry sector and so on. Ideally, having a large enough dataset of past security incidents, it would be possible to analyze it with automated tools and draw conclusions that may help in preventing future incidents. Unfortunately, it seems that there are only a few publicly available datasets of this kind that are of good quality. The paper reports our initial efforts in collecting all publicly available security incidents datasets, and building a single, large dataset that can be used to draw statistically significant observations. In order to argue about its statistical quality, we analyze the resulting combined dataset against the original ones. Additionally, we perform an analysis of the combined dataset and compare our results with the existing literature. Finally, we present our findings, discuss the limitations of the proposed approach, and point out interesting research directions.

show abstract

Section: Discussionmentioning

confidence: 99%

Merging Datasets of CyberSecurity Incidents for Fun and Insight

et al. 2021

View full text Add to dashboard Cite

show abstract

“…95 In the context of big data analysis, traditional 75 Figueiredo et al 20 + SVM, SVDD, KPCA, GKPCA ROC KPCA, GKPCA Figueiredo et al 20 AANN ROC -Figueiredo and Cross 76 NLPCA ROC -Oh et al 78 KPCA N/A -data cleaning working sequentially cannot easily be applied to ever-growing complicated structures. 96 Thus, the parallel execution of any method should be in line with the five big data quality standards discussed previously. The data cleaning process is always performed before initiating the subsequent ML processes.…”

Section: Data Cleaningmentioning

confidence: 93%

Machine learning and structural health monitoring overview with emerging technology and high-dimensional data source highlights

Malekloo

Özer

AlHamaydeh

et al. 2021

Structural Health Monitoring

262

View full text Add to dashboard Cite

Conventional damage detection techniques are gradually being replaced by state-of-the-art smart monitoring and decision-making solutions. Near real-time and online damage assessment in structural health monitoring (SHM) systems is a promising transition toward bridging the gaps between the past’s applicative inefficiencies and the emerging technologies of the future. In the age of the smart city, Internet of Things (IoT), and big data analytics, the complex nature of data-driven civil infrastructures monitoring frameworks has not been fully matured. Machine learning (ML) algorithms are thus providing the necessary tools to augment the capabilities of SHM systems and provide intelligent solutions for the challenges of the past. This article aims to clarify and review the ML frontiers involved in modern SHM systems. A detailed analysis of the ML pipelines is provided, and the in-demand methods and algorithms are summarized in augmentative tables and figures. Connecting the ubiquitous sensing and big data processing of critical information in infrastructures through the IoT paradigm is the future of SHM systems. In line with these digital advancements, considering the next-generation SHM and ML combinations, recent breakthroughs in (1) mobile device-assisted, (2) unmanned aerial vehicles, (3) virtual/augmented reality, and (4) digital twins are discussed at length. Finally, the current and future challenges and open research issues in SHM-ML conjunction are examined. The roadmap of utilizing emerging technologies within ML-engaged SHM is still in its infancy; thus, the article offers an outlook on the future of monitoring systems in assessing civil infrastructure integrity.

show abstract

“…As of now, there are two classifications of data cleansing: traditional data cleansing and data cleansing for big data. Traditional data cleansing techniques are so called because it is not used to manage massive volumes of data, such as Potter's Wheel and Intelliclean [12].…”

Section: Data-cleaning Applicationsmentioning

confidence: 99%

Sample Reduction for Physiological Data Analysis Using Principal Component Analysis in Artificial Neural Network

et al. 2021

View full text Add to dashboard Cite

With its potential, extensive data analysis is a vital part of biomedical applications and of medical practitioner interpretations, as data analysis ensures the integrity of multidimensional datasets and improves classification accuracy; however, with machine learning, the integrity of the sources is compromised when the acquired data pose a significant threat in diagnosing and analysing such information, such as by including noisy and biased samples in the multidimensional datasets. Removing noisy samples in dirty datasets is integral to and crucial in biomedical applications, such as the classification and prediction problems using artificial neural networks (ANNs) in the body’s physiological signal analysis. In this study, we developed a methodology to identify and remove noisy data from a dataset before addressing the classification problem of an artificial neural network (ANN) by proposing the use of the principal component analysis–sample reduction process (PCA–SRP) to improve its performance as a data-cleaning agent. We first discuss the theoretical background to this data-cleansing methodology in the classification problem of an artificial neural network (ANN). Then, we discuss how the PCA is used in data-cleansing techniques through a sample reduction process (SRP) using various publicly available biomedical datasets with different samples and feature sizes. Lastly, the cleaned datasets were tested through the following: PCA–SRP in ANN accuracy comparison testing, sensitivity vs. specificity testing, receiver operating characteristic (ROC) curve testing, and accuracy vs. additional random sample testing. The results show a significant improvement in the classification of ANNs using the developed methodology and suggested a recommended range of selectivity (Sc) factors for typical cleaning and ANN applications. Our approach successfully cleaned the noisy biomedical multidimensional datasets and yielded up to an 8% increase in accuracy with the aid of the Python language.

show abstract

A Review on Data Cleansing Methods for Big Data

Cited by 147 publications

References 6 publications

Merging Datasets of CyberSecurity Incidents for Fun and Insight

Merging Datasets of CyberSecurity Incidents for Fun and Insight

Machine learning and structural health monitoring overview with emerging technology and high-dimensional data source highlights

Sample Reduction for Physiological Data Analysis Using Principal Component Analysis in Artificial Neural Network

Contact Info

Product

Resources

About