Simple multi-party set reconciliation

Abstract

Many distributed cloud-based services use multiple loosely consistent replicas of user information to avoid the high overhead of more tightly coupled synchronization. Periodically, the information must be synchronized, or reconciled. One can place this problem in the theoretical framework of set reconciliation: two parties A1 and A2 each hold a set of keys, named S1 and S2 respectively, and the goal is for both parties to obtain S1 ∪ S2. Typically, set reconciliation is interesting algorithmically when sets are large but the set difference |S1 − S2| + |S2 − S1| is small. In this setting the focus is on accomplishing reconciliation efficiently in terms of communication; ideally, the communication should depend on the size of the set difference, and not on the size of the sets. In this paper, we extend recent approaches using Invertible Bloom Lookup Tables (IBLTs) for set reconciliation to the multi-party setting. There are three or more parties A1, A2, . . . , An holding sets of keys S1, S2, . . . , Sn respectively, and the goal is for all parties to obtain ∪iSi. Whiel this could be done by pairwise reconciliations, we seek more effective methods. Our general approach can function even if the number of parties is not exactly known in advance, and with some additional cost can be used to determine which other parties hold missing keys. Our methodology uses network coding techniques in conjunction with IBLTs, allowing efficiency in network utilization along with efficiency obtained by passing messages of size O(| ∪i Si − ∩iSi|). By connecting reconciliation with network coding, we can provide efficient reconciliation methods for a number of natural distributed settings.[8] use multiple loosely consistent replicas of user information because of the high overhead of keeping replicas synchronized at all times. Further, users often retain copies of their information on laptops, tablets, phones and Personal Digital Assistants (PDAs); these devices are often disconnected from cloud storage and thus can diverge from the corresponding copies in the cloud. The situation naturally grows even more complicated when multiple users have access to information, because the number of replicas can increase with the number of users. Periodically, however copies of information objects must be synchronized or reconciled. One can also view the need for reconciliation at a higher level, such as for loosely consistent replicas of large databases that may be used for availability by information providers.This paper focuses on the basic problem of set reconciliation. In the 2-party setting, two parties A 1 and A 2 respectively have (usually very similar) sets S 1 and S 2 , and want to reconcile so both have S 1 ∪ S 2 . Our major contribution is to extend the recent approach to set reconciliation for two parties using Invertible Bloom Lookup Tables (IBLTs) to the multi-party setting, where there are three or more parties holding sets S 1 , S 2 , S 3 , . . . , S n , and the goal is for all parties to obtain ∪ i S i . This could of course be do...

MCFsyn: A Multi-Party Set Reconciliation Protocol With the Marked Cuckoo Filter

“…We note that most-recently, the IBLT and characteristic polynomial are generalized from two-party set reconciliation to multi-party scenarios [16] [17]. Specifically, the binary fields in IBLT are extended to multi-ary to aggregate element information [16]. The XOR operations in the IBLT are also redefined in the multi-ary system correspondingly.…”

Robust Set Reconciliation via Locality Sensitive Hashing

“…Standard set reconciliation has been studied in the context of distributed synchronization, with many possible applications, see e.g. [21,29,10,23,25] and citations therein. As a fairly recent example, IBLTs (described below) have been proffered as a technique for scalable synchronization of transactions for Bitcoin, and have been discussed as an addition to the Bitcoin protocol [5].…”

Reconciling Graphs and Sets of Sets

“…These ideas have been extended, for example to multi-party reconciliation [8,24]. However, there remain many open problems related to reconciliation of objects beyond sets.…”