Improved single-round protocols for remote file synchronization

Abstract

Abstract-Given two versions of a file, a current version located on one machine and an outdated version known only to another machine, the remote file synchronization problem is how to update the outdated version over a network with a minimal amount of communication. In particular, when the versions are very similar, the total data transmitted should be significantly smaller than the file size. File synchronization problems arise in many application scenarios such as web site mirroring, file system backup and replication, and web access over slow links. An open source tool for this problem, called rsync and included in many Linux distributions, is widely used in such scenarios. rsync uses a single round of messages between the two machines. While recent research has shown that significant additional savings in bandwidth consumption are possible through the use of optimized multi-round protocols, there are many scenarios where multiple rounds are undesirable.In this paper, we study single-round protocols for file synchronization that offer significant improvements over rsync. Our main contribution is a new approach to file synchronization based on the use of erasure codes. Using this approach, we design a single-round protocol that is provably efficient with respect to common measures of file distance, and another optimized practical protocol that shows promising improvements over rsync on our data sets. In addition, we show how to obtain some improvements by engineering the rsync approach, and study the use of recent set reconciliation techniques in file synchronization protocols.

DeltaNC: Efficient File Updates for Network-Coding-Based Cloud Storage Systems

“…Irmak et. al proposed a single-round protocol based on rsync, an open source tool for file synchronization, that utilizes erasure codes to reduce network traffic [14].…”

An update model for network coding in cloud storage systems

“…Utilizing Reed-Solomon codes for cloud storage, the content of a source data is divided into k blocks of a same size, and then n > k encoded blocks are generated by the source node, such that any k out of n encoded blocks can be used to recover the original file. To minimize the computational and communication cost of the file updates, the delta update of erasure codes have been examined and proposed in [15], [16]. Chesterfield et.…”

“…al addressed the update problem in wireless broadcast channels using hierarchical decomposable homomorphic erasure hashes [15]. The rateless erasure encoder over Galois field GF (2 16 ) allows the receivers to use any received hash in place of any missing one. The proposed system conserves battery power while minimizing the synchronization time of the mobile devices.…”

“…Irmak et. al proposed a single-round protocol based on rsync, an open source tool for file synchronization, that utilizes erasure code over compressed delta data to reduce network traffic [16].…”

“…Furthermore, we provide the corresponding update algorithm based on this model. In contrast to the delta update of erasure codes [15], [16], DUM applies the update directly to coded blocks without decoding any of them.…”