An Advanced Big Data Quality Framework Based on Weighted Metrics

Abstract

While big data benefits are numerous, the use of big data requires, however, addressing new challenges related to data processing, data security, and especially degradation of data quality. Despite the increased importance of data quality for big data, data quality measurement is actually limited to few metrics. Indeed, while more than 50 data quality dimensions have been defined in the literature, the number of measured dimensions is limited to 11 dimensions. Therefore, this paper aims to extend the measured dimensions by defining four new data quality metrics: Integrity, Accessibility, Ease of manipulation, and Security. Thus, we propose a comprehensive Big Data Quality Assessment Framework based on 12 metrics: Completeness, Timeliness, Volatility, Uniqueness, Conformity, Consistency, Ease of manipulation, Relevancy, Readability, Security, Accessibility, and Integrity. In addition, to ensure accurate data quality assessment, we apply data weights at three data unit levels: data fields, quality metrics, and quality aspects. Furthermore, we define and measure five quality aspects to provide a macro-view of data quality. Finally, an experiment is performed to implement the defined measures. The results show that the suggested methodology allows a more exhaustive and accurate big data quality assessment, with a more extensive methodology defining a weighted quality score based on 12 metrics and achieving a best quality model score of 9/10.

Architecting Data-Driven Applications: From Architectural Design to the Data Quality

“…DAFQA can handle all data formats and creates Python code to verify the six primary data quality dimensions. In [41] proposed a Big Data Quality Assessment Framework with 12 metrics, including four new ones: Integrity, Accessibility, Ease of Manipulation, and Security, and measured data quality with weights at three levels and five quality aspects for a macro view. In [107] proposed…”

Quality Anomaly Detection Using Predictive Techniques: An Extensive Big Data Quality Framework for Reliable Data Analysis

“…This metric refers to data respecting the expected rules and constraints regarding data type [16]. Thus, a conform dataset means that all data values in a particular column or field have the same data type and format.…”

“…As misspelled words can make it difficult for users to search, filter, or analyze the data accurately. The ability to extract information from data is called data readability [16]. Thus, data readability patterns should highlight how much data is readable and represent meaningful insights.…”

Structured dataset of human-machine interactions enabling adaptive user interfaces

“…To validate the resulting dataset (sequences_df_prep_EN.csv) and facilitate the reuse, a data profiling analysis was conducted. First, it was important to guarantee the consistency, which could be defined as data presented in the same standard structure and its correctness in relationship with other data 16 . Therefore, it was validated that www.nature.com/scientificdata www.nature.com/scientificdata/ the elements within the interaction sequences correspond to the elements on the UI JSON file, and the users' IDs exist in the users.csv file.…”