Distributed implementations of dependency discovery algorithms

Abstract

We analyze the problem of discovering dependencies from distributed big data. Existing (non-distributed) algorithms focus on minimizing computation by pruning the search space of possible dependencies. However, distributed algorithms must also optimize communication costs, especially in shared-nothing settings, leading to a more complex optimization space. To understand this space, we introduce six primitives shared by existing dependency discovery algorithms, corresponding to data processing steps separated by communication barriers. Through case studies, we show how the primitives allow us to analyze the design space and develop communication-optimized implementations. Finally, we support our analysis with an experimental evaluation on real datasets.

Efficient distributed discovery of bidirectional order dependencies

“…In this section, we evaluate DISTOD's performance in different settings and on various datasets. We compare its runtime with all existing complete OD discovery algorithms, which are FASTOD-BID [25] and its distributed variant DIST-FASTOD-BID [22]. Note that ORDER [15], its hybrid variant [12], and OCDDISCOVER [5] produce incomplete results and are, therefore, not comparable to our approach.…”

“…The motivation for this research project is the observation that most distributed data profiling algorithms, including [14,21,22,33], are built on top of dataflow-based distributed computing frameworks, such as Apache Spark [31] or Apache Flink [30]. These frameworks force the discovery algorithms into batch processing, which is an unsuitable paradigm for all known dependency discovery approaches, because they rely on dynamic pruning and dynamic candidate generation tech- niques.…”

“…Distributed order dependency discovery Because FASTOD-BID is the only complete and correct OD algorithm, not much research exists on distributed OD discovery. In [22], Saxena et al proposed common map-reduce style primitives (based on Apache Spark) into which they could break down any existing data profiling algorithm. In this way, they presented distributed versions of different dependency discovery algorithms including FASTOD-BID-we call this implementation DIST-FASTOD-BID.…”

“…This has the advantage that we can discard the attribute type information and all concrete values, which saves memory and-because the computations effectively deal with integers only-makes the operations on partitions fast and simple. Stripped partitions DISTOD requires only the sorted partitions for each {A} ∈ R (level l 1 of the candidate lattice); for attribute sets with |X | > 1 (higher levels), it replaces sorted partitions with the smaller stripped partitions [5,10,12,22,25,26] (also known as position list indexes [2,15,19,20]).…”

PatchIndex: exploiting approximate constraints in distributed databases

“…An example problem is the discovery of unique column combinations (UCC), which is a set of columns whose projection only contains unique rows. Finding all exact UCCs for a given relation is shown to be NP-hard [8] and various algorithms were introduced in [9,19,22,23]. These algorithms mainly focus on the discovery of exact UCCs, so unclean data might hamper the discovery.…”