Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes

Abstract

Complex data pipelines are increasingly common in diverse applications such as BI reporting and ML modeling. These pipelines often recur regularly (e.g., daily or weekly), as BI reports need to be refreshed, and ML models need to be retrained. However, it is widely reported that in complex production pipelines, upstream data feeds can change in unexpected ways, causing downstream applications to break silently that are expensive to resolve.Data validation has thus become an important topic, as evidenced by notable recent efforts from Google and Amazon, where the objective is to catch data quality issues early as they arise in the pipelines. Our experience on production data suggests, however, that on string-valued data, these existing approaches yield high false-positive rates and frequently require human intervention. In this work, we develop a corpus-driven approach to auto-validate machine-generated data by inferring suitable data-validation "patterns" that accurately describe the underlying data-domain, which minimizes false-positives while maximizing data quality issues caught. Evaluations using production data from real data lakes suggest that Auto-Validate is substantially more effective than existing methods. Part of this technology ships as an Auto-Tag feature in Microsoft Azure Purview.

Data Lakes: A Survey of Functions and Systems

“…In [137], Song et al have tackled a specific data cleaning problem, i.e., data validation. In a large enterprise data lake with terabytes of data, the data may change with time.…”

“…The data validation rules indicate whether the changes are significant enough, and will affect the downstream applications. The approach in [137] tries to automatically derive such rules from the machine-generated, string-valued data, rather than inferred by human experts. In principle, it formulates the rule inference problem as an optimization problem, which balances between false-positive-rate minimization and quality issue preserving.…”

Desbordante: from benchmarking suite to high-performance science-intensive data profiler (preprint)

“…Such tiny unit tests for data validation can be embedded into a workflow and immediately raise a flag if anything unexpected happens, e.g., newly arrived data violates a primitive that was described within the Great Expectations framework. A similar idea is used in the Auto-Validate [32] system.…”

Semantic programming by example with pre-trained models

“…The goal of string profiling is to learn succinct regular expression patterns that describe a collection of strings. These profiles are useful for a myriad of applications, from checking the quality of data, computing the syntactic similarity between strings, tagging large datasets with column metadata [Song and He 2021] and making string transformation synthesizers more robust by improving the ranking of programs [Ellis and Gulwani 2017] or learning separate programs for examples with different profiles [Padhi et al 2018].…”