An Automated Big Data Accuracy Assessment Tool

Mylavarapu, Goutam; Thomas, Johnson; Viswanathan, Kartik

doi:10.1109/icbda.2019.8713218

Cited by 14 publications

(10 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data Accuracy refers to the reliability and the correctness of the information contained within data. It measures the closeness of the collected data to the actual or expected value [28]. Examples of inaccurate values can be an age that exceeds 120 or a birth date in a future date, as these values are unusual and outside of the expected scale for these attributes.…”

Section: ) Accuracymentioning

confidence: 99%

Quality Anomaly Detection Using Predictive Techniques: An Extensive Big Data Quality Framework for Reliable Data Analysis

Widad,

Saida,

Gahi

2023

IEEE Access

View full text Add to dashboard Cite

The increasing reliance on Big Data analytics has highlighted the critical role of data quality in ensuring accurate and reliable results. Consequently, organizations aiming to leverage the power of Big Data recognize the crucial role of data quality as an integral component. One notable type of data quality anomaly observed in big datasets is the presence of outlier values. Detecting and addressing these outliers have become a subject of interest across diverse domains, leading to the development of numerous anomaly detection approaches. Although anomaly detection has witnessed a proliferation of practices in recent years, a significant gap remains in addressing anomalies related to the other aspects of data quality. Indeed, while most approaches focus on identifying anomalies that deviate from the expected patterns, they do not consider irregularities in data quality, such as missing, incorrect, or inconsistent data. Moreover, most of approaches are domain-correlated and lack the capability to detect anomalies in a generic manner. Thus, we aim through this paper to address this gap in the field and provide a holistic and effective solution for Big Data quality anomaly detection. To achieve this, we suggest a novel approach that allows a comprehensive detection of Big Data quality anomalies related to six quality dimensions: Accuracy, Consistency, Completeness, Conformity, Uniqueness, and Readability. Moreover, the framework allows for sophisticated detection of generic data quality anomalies through the implementation of an intelligent anomaly detection model without any correlation to a specific field. Furthermore, we introduce and measure a new metric called "Quality Anomaly Score," which refers to the degree of anomalousness of the quality anomalies of each quality dimension and the entire dataset. Through the implementation and evaluation of our framework, the suggested framework has achieved an accuracy score of up to 99.91% and an F-score of 98.07%.

show abstract

Section: ) Accuracymentioning

confidence: 99%

Quality Anomaly Detection Using Predictive Techniques: An Extensive Big Data Quality Framework for Reliable Data Analysis

Widad,

Saida,

Gahi

2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Despite possible delays in dissemination, the value of these products for improving the understanding of ecosystem and convective processes and aerosol properties; quantifying energy, water, and carbon cycle fluxes and their trends; and developing and validating AI/ML or physically based models have been demonstrated in numerous studies (e.g., Jung et al 2019;Ojha et al 2021;Shiklomanov et al 2021). ML techniques have also demonstrated their value, including for automated data QA/QC and processing, estimating data at locations or temporal periods outside the observation window, and for the evaluation of short-and long-term behaviors (e.g., Mylavarapu, Thomas, and Viswanathan 2019;Okafor, Alghorani, and Delaney 2020;Sanhudo, Rodrigues, and Filho 2021). Yet, the resolution, coverage, and diversity of the existing products; the level of automation to generate/update them; and the potential use of physicalbased models to guide their development constitute areas where improvements are particularly needed.…”

Section: Daq For Developing Training and Test Datasetsmentioning

confidence: 99%

Artificial Intelligence for Earth System Predictability (AI4ESP) (2021 Workshop Report)

Hickmon¹,

Varadharajan²,

Hoffman³

et al. 2022

View full text Add to dashboard Cite

“…Both frameworks do not support data acquisition and contextual level data quality analytics. In most of the literature, the data quality accuracy metric has been widely focused [34]. Such as in the article [35], authors analyze the multivariate anomalies with principal component analysis and compute the accuracy of the healthcare dataset that contain the vital sign of patients who were admitted in the intensive care unit.…”

Section: Background and Related Workmentioning

confidence: 99%

Web Objects Based Contextual Data Quality Assessment Model for Semantic Data Application

Jarwar

Chong²

2020

Applied Sciences

View full text Add to dashboard Cite

Due to the convergence of advanced technologies such as the Internet of Things, Artificial Intelligence, and Big Data, a healthcare platform accumulates data in a huge quantity from several heterogeneous sources. The adequate usage of this data may increase the impact of and improve the healthcare service quality; however, the quality of the data may be questionable. Assessing the quality of the data for the task in hand may reduce the associated risks, and increase the confidence of the data usability. To overcome the aforementioned challenges, this paper presents the web objects based contextual data quality assessment model with enhanced classification metric parameters. A semantic ontology of virtual objects, composite virtual objects, and services is also proposed for the parameterization of contextual data quality assessment of web objects data. The novelty of this article is the provision of contextual data quality assessment mechanisms at the data acquisition, assessment, and service level for the web objects enabled semantic data applications. To evaluate the proposed data quality assessment mechanism, web objects enabled affective stress and teens’ mood care semantic data applications are designed, and a deep data quality learning model is developed. The findings of the proposed approach reveal that, once a data quality assessment model is trained on web objects enabled healthcare semantic data, it could be used to classify the incoming data quality in various contextual data quality metric parameters. Moreover, the data quality assessment mechanism presented in this paper can be used to other application domains by incorporating data quality analysis requirements ontology.

show abstract

An Automated Big Data Accuracy Assessment Tool

Cited by 14 publications

References 8 publications

Quality Anomaly Detection Using Predictive Techniques: An Extensive Big Data Quality Framework for Reliable Data Analysis

Quality Anomaly Detection Using Predictive Techniques: An Extensive Big Data Quality Framework for Reliable Data Analysis

Artificial Intelligence for Earth System Predictability (AI4ESP) (2021 Workshop Report)

Web Objects Based Contextual Data Quality Assessment Model for Semantic Data Application

Contact Info

Product

Resources

About