Discovery of genuine functional dependencies from relational data with missing values

Berti‐Équille, Laure; Harmouch, Hazar; Naumann, Felix; Novelli, Noël; Thirumuruganathan, Saravanan

doi:10.14778/3204028.3204032

Cited by 39 publications

(18 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Functional dependency (FD) is an orthogonal approach that leverages multicolumn dependency for data quality [56]. While FDs inferred from individual table instances often may not hold in a semantic sense [19,51], we nevertheless evaluate the fraction of benchmark columns that are part of any FD from their original tables, which would be a recall upper-bound for FD-based approaches. For simplicity, in this analysis, we assume a perfect precision for FD-based methods.…”

Section: Methods Comparedmentioning

confidence: 99%

Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes

Song

2021

Proceedings of the 2021 International Conference on Management of Data

View full text Add to dashboard Cite

Complex data pipelines are increasingly common in diverse applications such as BI reporting and ML modeling. These pipelines often recur regularly (e.g., daily or weekly), as BI reports need to be refreshed, and ML models need to be retrained. However, it is widely reported that in complex production pipelines, upstream data feeds can change in unexpected ways, causing downstream applications to break silently that are expensive to resolve.Data validation has thus become an important topic, as evidenced by notable recent efforts from Google and Amazon, where the objective is to catch data quality issues early as they arise in the pipelines. Our experience on production data suggests, however, that on string-valued data, these existing approaches yield high false-positive rates and frequently require human intervention. In this work, we develop a corpus-driven approach to auto-validate machine-generated data by inferring suitable data-validation "patterns" that accurately describe the underlying data-domain, which minimizes false-positives while maximizing data quality issues caught. Evaluations using production data from real data lakes suggest that Auto-Validate is substantially more effective than existing methods. Part of this technology ships as an Auto-Tag feature in Microsoft Azure Purview.

show abstract

Section: Methods Comparedmentioning

confidence: 99%

Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes

Song

2021

Proceedings of the 2021 International Conference on Management of Data

View full text Add to dashboard Cite

show abstract

“…The above condition allows an FD to hold for most tuples allowing some violations. This equation captures the essence of approximate FDs used in multiple works [3,19,20,25,30]. Two core approaches are adopted in these works to discover approximate dependencies that satisfy:…”

Section: Functional Dependenciesmentioning

confidence: 99%

“…(1) Use likelihood-based measures to find groups of attributes that satisfy Equation 1 [3,19,20,25]. Typically these methods compute the approximate distribution (and likelihood) by considering co-occurrence counts between values of (X, Y ) and normalizing those by counts of values of X [3,20,25].…”

Section: Functional Dependenciesmentioning

confidence: 99%

A Statistical Perspective on Discovering Functional Dependencies in Noisy Data

Zhang

Guo

Ρεκατσίνας

2020

Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

We study the problem of discovering functional dependencies (FD) from a noisy data set. We adopt a statistical perspective and draw connections between FD discovery and structure learning in probabilistic graphical models. We show that discovering FDs from a noisy data set is equivalent to learning the structure of a model over binary random variables, where each random variable corresponds to a functional of the data set attributes. We build upon this observation to introduce FDX a conceptually simple framework in which learning functional dependencies corresponds to solving a sparse regression problem. We show that FDX can recover true functional dependencies across a diverse array of realworld and synthetic data sets, even in the presence of noisy or missing data. We find that FDX scales to large data instances with millions of tuples and hundreds of attributes while it yields an average F 1 improvement of 2× against state-of-the-art FD discovery methods.

show abstract

“…We devise an effective and efficient algorithm to automatically discover PFDs from dirty datasets. Note that, although profiling ICs from clean data [14,4] has been widely studied, discovering them from dirty data is known to be much harder [6,9]. (Section 4) 4.…”

Section: Contributionsmentioning

confidence: 99%

Pattern functional dependencies for data cleaning

Qahtan

Tang²,

Ouzzani³

et al. 2020

Proc. VLDB Endow.

View full text Add to dashboard Cite

Patterns (or regex-based expressions) are widely used to constrain the format of a domain (or a column), e.g., a Year column should contain only four digits, and thus a value like "1980-" might be a typo. Moreover, integrity constraints (ICs) defined over multiple columns, such as (conditional) functional dependencies and denial constraints, e.g., a ZIP code uniquely determines a city in the UK, have been widely used in data cleaning. However, a promising, but not yet explored, direction is to combine regex-and IC-based theories to capture data dependencies involving partial attribute values. For example, in an employee ID such as"F-9-107", "F" is sufficient to determine the finance department. Inspired by the above observation, we propose a novel class of ICs, called pattern functional dependencies (PFDs), to model fine-grained data dependencies gleaned from partial attribute values. These dependencies cannot be modeled using traditional ICs, such as (conditional) functional dependencies, which work on entire attribute values. We also present a set of axioms for the inference of PFDs, analogous to Armstrong's axioms for FDs, and study the complexity of consistency and implication analysis of PFDs. Moreover, we devise an effective algorithm to automatically discover PFDs even in the presence of errors in the data. Our extensive experiments on 15 real-world datasets show that our approach can effectively discover valid and useful PFDs over dirty data, which can then be used to detect data errors that are hard to capture by other types of ICs.

show abstract

Discovery of genuine functional dependencies from relational data with missing values

Cited by 39 publications

References 31 publications

Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes

Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes

A Statistical Perspective on Discovering Functional Dependencies in Noisy Data

Pattern functional dependencies for data cleaning

Contact Info

Product

Resources

About