Efficient order dependency detection

Langer, Philipp; Naumann, Felix

doi:10.1007/s00778-015-0412-3

Cited by 36 publications

(66 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This complexity is similar to previous FD and inclusion dependency discovery algorithms such as TANE [10]. In one of our experimental runs, using a flight dataset with 1000 tuples and 20 attributes, our algorithm takes under one second whereas the algorithm from [13] does not terminate after five hours (Section 5).…”

Section: Problem Statement and Contributionssupporting

confidence: 59%

“…Our pruning rules do not affect the completeness of the discovered ODs. The list-based algorithm in [13] uses aggressive pruning rules to overcome their factorial complexity, which we show leads to incompleteness (counter to the claim of complete discovery made in [13]). They can miss important ODs.…”

Section: Problem Statement and Contributionsmentioning

confidence: 84%

“…We do not consider bidirectional ODs in this paper, which allow a mix of ascending and descending (desc) orders [25]. We also only consider lexicographic order specifications, as per the SQL order-by semantics, and do not consider pointwise ODs [7,8] [13,24,25] than working with pointwise ODs, because the sequence of the attributes in an order specification as in the order-by statement matters. Figure 1 shows a sound and complete axiomatization for ODs [22].…”

Section: Framework and Backgroundmentioning

confidence: 99%

“…Indeed, we provide a polynomial mapping of list-based ODs into equivalent set-based canonical ODs. The mapping allows us to develop an efficient OD discovery algorithm that traverses a much smaller set-containment lattice rather than a listcontainment lattice used in prior work [13].…”

Section: Mapping To Canonical Formmentioning

confidence: 99%

“…FASTOD traverses a lattice of all possible sets of attributes in a level-wise manner ( Figure 3) since list-based ODs can be mapped into the equivalent set-based ODs (Theorem 5). In contrast, the OD discovery algorithm from [13] traverses a lattice of all possible lists of attributes, which leads to factorial time complexity. In level L l , our algorithm generates candidate ODs with l attributes using com-puteODs(L l ).…”

Section: Fastod Main Algorithmmentioning

confidence: 99%

See 4 more Smart Citations

Effective and complete discovery of order dependencies via set-based axiomatization

et al. 2017

View full text Add to dashboard Cite

Integrity constraints (ICs) provide a valuable tool for expressing and enforcing application semantics. However, formulating constraints manually requires domain expertise, is prone to human errors, and may be excessively time consuming, especially on large datasets. Hence, proposals for automatic discovery have been made for some classes of ICs, such as functional dependencies (FDs), and recently, order dependencies (ODs). ODs properly subsume FDs, as they can additionally express business rules involving order; e.g., an employee never has a higher salary while paying lower taxes compared with another employee.We address the limitations of prior work on OD discovery which has factorial complexity in the number of attributes, is incomplete (i.e., it does not discover valid ODs that cannot be inferred from the ones found) and is not concise (i.e., it can result in "redundant" discovery and overly large discovery sets). We improve significantly on complexity, offer completeness, and define a compact canonical form. This is based on a novel polynomial mapping to a canonical form for ODs, and a sound and complete set of axioms (inference rules) for canonical ODs. This allows us to develop an efficient set-containment, lattice-driven OD discovery algorithm that uses the inference rules to prune the search space. Our algorithm has exponential worst-case time complexity, O(2 |R| ), in the number of attributes and linear complexity in the number of tuples. We prove that it produces a complete, minimal set of ODs (i.e., minimal with regards to the canonical representation). Finally, using real and synthetic datasets, we experimentally show orders-of-magnitude performance improvements over the current state-of-the-art algorithm and demonstrate effectiveness of our techniques.

show abstract

Section: Problem Statement and Contributionssupporting

confidence: 59%

Section: Problem Statement and Contributionsmentioning

confidence: 84%