Extending Conditional Dependencies with Built-in Predicates

Abstract

This paper proposes a natural extension of conditional functional dependencies (CFDs [22]) and conditional inclusion dependencies (CINDs [30]), denoted by CFD p s and CIND p s, respectively, by specifying patterns of data values with ̸ =, <, ≤, > and ≥ predicates. As data quality rules, CFD p s and CIND p s are able to capture errors that commonly arise in practice but cannot be detected by CFDs and CINDs. We establish two sets of results for central technical problems associated with CFD p s and CIND p s. (a) One concerns the satisfiability and implication problems for CFD p s and CIND p s, taken separately or together. These are important for, e.g., deciding whether data quality rules are dirty themselves, and for removing redundant rules. We show that despite the increased expressive power, the static analyses of CFD p s and CIND p s retain the same complexity as their CFDs and CINDs counterparts. (b) The other concerns validation of CFD p s and CIND p s. We show that given a set Σ of CFD p s and CIND p s on a database D, a set of SQL queries can be automatically generated that, when evaluated against D, return all tuples in D that violate some dependencies in Σ. We also experimentally verified the efficiency and effectiveness of our SQL based error detection techniques, using real-life data. This provides commercial DBMS with an immediate capability to detect errors based on CFD p s and CIND p s.

One-Pass Inconsistency Detection Algorithms for Big Data

“…As an important problem in data quality management, the inconsistency problem including both the detecting and repairing problem has been well studied [2], [7], [8], [11], [5], [12]- [16]. Some researchers use statistical model and propose some thresholds to identify the inconsistencies [17], [18].…”

“…Some researchers use statistical model and propose some thresholds to identify the inconsistencies [17], [18]. Other researchers use constraints such as FDs and CFDs [2], [7], [8], [10]- [13], [19] to detect inconsistencies. In order to make the rules more general and more expressive, denial constraints (DCs) are also regarded as rules.…”

“…In [7] and [8], researchers proposed a SQL-based automatical detection method to identify the tuples violating CFDs. In [7], they get a tableaux merged by multiple CFDs, and translate that into a single pair of SQL queries.…”

Easy First Relation Extraction with Information Redundancy

“…Data dependencies have well studied for improving data quality (Ma, 2011;Ma et al, 2015;Fan and Geerts, 2012), which essentially make use of data redundancies and dependencies. Our approach is partially inspired by these studies.…”