Improving distributed join efficiency with extended bloom filter operations

Abstract

Bloom filter based algorithms have proven successful as very efficient technique to reduce communication costs of database joins in a distributed setting. However, the full potential of bloom filters has not yet been exploited. Especially in the case of multi-joins, where the data is distributed among several sites, additional optimization opportunities arise, which require new bloom filter operations and computations. In this paper, we present these extensions and point out how they improve the performance of such distributed joins. While the paper focuses on efficient join computation, the described extensions are applicable to a wide range of usages, where bloom filters are facilitated for compressed set representation.

BSI: Bloom Filter-Based Semantic Indexing for Unstructured P2P Networksh

“…These bit arrays are similar to those employed in traditional Bloom filters and is supported by a sufficiently large body of research work [14], [16], [17] that allows us to estimate number of documents reachable for a multi-concept query solely based on these bit arrays. Similar to level 1, level 2(TSBF 2,P ) also contains multiple bit arrays each representing different multi-concept queries that whose concepts have C as the least common ancestor in the ontology hierarchy for which P has at least one qualified document in its local document collection (TSBF 2,P (C)).…”

“…Research community has proposed many works to estimate the cardinality(i.e. number of elements) of an original set solely based on its Bloom filter bit array [14], [16], [17]. For our work we used the work presented by authors of [16].…”

PCIR: Combining DHTs and peer clusters for efficient full-text P2P indexing

“…8), we can estimate the number of true bits in the Bloom filter of the intersection of the two sets A ∩ B. From there, we can use a theorem from [42] to estimate the number of elements in the intersection.…”

“…Thus we can use Lemma 4.1 from [42] to estimate the number of objects hashed into it: |SET BF ∩ | = m * 1 − (1 − 1/m) kn , where n = E(|A ∩ B|). Combining that with equation 10, we get an estimation for E(|A ∩ B|):…”

Join processing using Bloom filter in MapReduce

“…The bloomjoin algorithm is combined with a group-by operation and extended to multi-way joins in [10]. More recent studies [14,18] optimizes complex distributed multiway joins using this algorithm. However, while all these work assumes the join relations are not partitioned, in MapReduce environment, a dataset are split and distributed in many nodes.…”