Set Reconciliation via Counting Bloom Filters

Abstract

In this paper, we study the set reconciliation problem, in which each member of a node pair has a set of objects and seeks to deliver its unique objects to the other member. How could each node compute the set difference, however, is challenging in the set reconciliation problem. To address such an issue, we propose a lightweight but efficient method that only requires the pair of nodes to represent objects using a counting Bloom filter (CBF) of size OðdÞ and exchange with each other, where d denotes the total size of the set differences. A receiving node then subtracts the received CBF from its local one via minus operation proposed in this paper. The resultant CBF can approximately represent the union of the set differences and thus the set difference to each node can be identified after querying the resultant CBF. In this paper, we propose a novel estimator through which each node can accurately estimate not only the value of d but also the size of the set difference to each node. Such an estimation result can be used to optimize the parameter setting of the CBF to achieve less false positives and false negatives. Comprehensive analysis and evaluation demonstrates that our method is more efficient than prior BF-based methods in terms of achieving the same accuracy with less communication cost. Moreover, our reconciliating method needs no prior context logs and it is very useful in networking and distributed applications.

A Survey of Algorithms to Efficiently Reconcile Sets of Infomation

“…Recall that in the previous two sections, the protocols discussed guarantee recovery of S A ∪ S B whenever |(S A \ S B ) ∪ (S B \ S A )| ≤ d. In contrast, the Bloom filter approach allows recovery of S A ∪ S B with high probability whenever |(S A \ S B ) ∪ (S B \ S A )| ≤ d. For the remainder of this section, we discuss a variation of the protocol from [5] which uses the invertible Bloom filter first discussed in [6]. Similar ideas were also used in [3] and [7].…”

“…Section 3. describes a method for set reconciliation that leverages polynomial interpolation as in [10]. In Section 4. we describe an algorithm for set reconciliation that uses Bloom filter structures [5], [6], [7]. Lastly, in Section 5., we conclude this survey by summarizing ongoing work and identifying potential directions for future research.…”

Robust set reconciliation

“…The problem gets more challenging when using the edit distance [5], which is more useful when synchronizing two text files. For unordered data (i.e., sets) with the standard set-difference measure (or equivalently the Jaccard similarity), the problem is exactly the set reconciliation problem [7,18,9], which was mentioned at the beginning of the paper. For our problem, the elements in the sets are points in the Euclidean space and the distance measure is the EMD.…”

Page replacement algorithm based on counting bloom filter for NAND flash memory

“…In order to minimize the false positive rate, K and M should be selected correctly. According to [18], it can be concluded that the relationship between M and K can be denoted as:…”