Efficient denial constraint discovery with hydra

Abstract

Denial constraints (DCs) are a generalization of many other integrity constraints (ICs) widely used in databases, such as key constraints, functional dependencies, or order dependencies. Therefore, they can serve as a unified reasoning framework for all of these ICs and express business rules that cannot be expressed by the more restrictive IC types. The process of formulating DCs by hand is difficult, because it requires not only domain expertise but also database knowledge, and due to DCs' inherent complexity, this process is tedious and error-prone. Hence, an automatic DC discovery is highly desirable: we search for all valid denial constraints in a given database instance. However, due to the large search space, the problem of DC discovery is computationally expensive. We propose a new algorithm Hydra, which overcomes the quadratic runtime complexity in the number of tuples of state-of-the-art DC discovery methods. The new algorithm's experimentally determined runtime grows only linearly in the number of tuples. This results in a speedup by orders of magnitude, especially for datasets with a large number of tuples. Hydra can deliver results in a matter of seconds that to date took hours to compute.

Distributed implementations of dependency discovery algorithms

“…Hybrid algorithms switch back and forth between schemadriven and data-driven phases; examples include HyFD [14] for FDs, HyUCC [15] for UCCs and Hydra [4] for DCs. HyFD starts with a data-driven phase, but it generates evidence sets only from a sample of tuples.…”

“…At some point, HyFD may switch back to a data-driven phase and generate evidence sets from a different sample of tuples, and so on. HyUCC and Hydra [4] are similar to HyFD but Hydra switches only once from the data-driven phase to the schema-driven phase.…”

“…As discussed in Section 2.2, in UCC and FD discovery, this primitive can return true or false by comparing the counts of |πX | and |πY |, whereas in OD discovery, it needs to check the ordering of tuples within matching equivalence classes. 4. Join (join(Si, Sj, p)): This primitive joins two sets of elements, Si and Sj, using p as the join predicate.…”

“…First, a map-reduce job implements a self-join that joins pairs of tuples within the same equivalence class. Returning to Figure 1(a), equivalence classes for A generate tuple pairs (1,3) and (2,4); equivalence classes for B generate (3,4), and so on. To implement this in a distributed fashion, we use the Dis-Dedup + algorithm from [5].…”

“…As outlined in Section 2.2, HyFD alternates between data-driven and schema-driven phases. We note that HyUCC [15] and Hydra [4] can also be implemented using the plans described in this section, with some modifications: HyUCC uses different pruning rules for UCCs, whereas Hydra discovers DCs and hence uses a richer predicate space. Unlike HyFD, Hydra switches from the datadriven phase to the schema-driven phase only once, after the rate of generating DC violations drops below a user-supplied threshold.…”