Data Quality Identification Model for Power Big Data

Zheng, Haijie; Teng, Bing; Liu, Xiaobao; Zhang, Wenbin; Liu, Shenqi

doi:10.1007/978-981-19-5209-8_2

Cited by 3 publications

(2 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The performance of the framework was illustrated in terms of the false-positive rate and percentage of error in a dataset; when the error percentage increased in the dataset, the false-positive rate increased too. Zheng et al [84] proposed a data quality identification model (tri-training) to detect global outliers in batch processing. A power consumption dataset collected from various industries was used to evaluate the proposed model, and the results were compared to those achieved with existing methods in terms of error rates.…”

Section: Unclassified Techniquesmentioning

confidence: 99%

Cleaning Big Data Streams: A Systematic Literature Review

2023

View full text Add to dashboard Cite

In today’s big data era, cleaning big data streams has become a challenging task because of the different formats of big data and the massive amount of big data which is being generated. Many studies have proposed different techniques to overcome these challenges, such as cleaning big data in real time. This systematic literature review presents recently developed techniques that have been used for the cleaning process and for each data cleaning issue. Following the PRISMA framework, four databases are searched, namely IEEE Xplore, ACM Library, Scopus, and Science Direct, to select relevant studies. After selecting the relevant studies, we identify the techniques that have been utilized to clean big data streams and the evaluation methods that have been used to examine their efficiency. Also, we define the cleaning issues that may appear during the cleaning process, namely missing values, duplicated data, outliers, and irrelevant data. Based on our study, the future directions of cleaning big data streams are identified.

show abstract

Section: Unclassified Techniquesmentioning

confidence: 99%

Cleaning Big Data Streams: A Systematic Literature Review

2023

View full text Add to dashboard Cite

show abstract

“…They call for further research to enhance the model's core concepts, integrate it with other data processing technologies, and expand its application scope. The authors in [21] have proposed a data outliers identification model designed explicitly for power big data to address outliers in big data. The model uses data augmentation technology to group and map power data into different feature spaces.…”

Section: Related Workmentioning

confidence: 99%

An Automated Big Data Quality Anomaly Correction Framework Using Predictive Analysis

Elouataoui,

El Mendili,

Gahi

2023

Data

View full text Add to dashboard Cite

Big data has emerged as a fundamental component in various domains, enabling organizations to extract valuable insights and make informed decisions. However, ensuring data quality is crucial for effectively using big data. Thus, big data quality has been gaining more attention in recent years by researchers and practitioners due to its significant impact on decision-making processes. However, existing studies addressing data quality anomalies often have a limited scope, concentrating on specific aspects such as outliers or inconsistencies. Moreover, many approaches are context-specific, lacking a generic solution applicable across different domains. To the best of our knowledge, no existing framework currently automatically addresses quality anomalies comprehensively and generically, considering all aspects of data quality. To fill the gaps in the field, we propose a sophisticated framework that automatically corrects big data quality anomalies using an intelligent predictive model. The proposed framework comprehensively addresses the main aspects of data quality by considering six key quality dimensions: Accuracy, Completeness, Conformity, Uniqueness, Consistency, and Readability. Moreover, the framework is not correlated to a specific field and is designed to be applicable across various areas, offering a generic approach to address data quality anomalies. The proposed framework was implemented on two datasets and has achieved an accuracy of 98.22%. Moreover, the results have shown that the framework has allowed the data quality to be boosted to a great score, reaching 99%, with an improvement rate of up to 14.76% of the quality score.

show abstract