Co-Design and Verification of an Available File System

Abstract

Abstract. Distributed file systems play a vital role in large-scale enterprise services. However, the designer of a distributed file system faces a vexing choice between strong consistency and asynchronous replication. The former supports a standard sequential model by synchronising operations, but is slow and fragile. The latter is highly available and responsive, but exposes users to concurrency anomalies. In this paper, we describe a rigorous and general approach to navigating this trade-off by leveraging static verification tools that allow to verify different file system designs. We show that common file system operations can run concurrently without synchronisation, while still retaining a semantics reasonably similar to Posix hierarchical structure. The one exception is the move operation, for which we prove that, unless synchronised, it will have an anomalous behaviour.

A Highly-Available Move Operation for Replicated Trees

“…Najafzadeh et al [5], [6] previously implemented a replicated filesystem with a move operation, and analysed the case of concurrent move operations introducing a cycle. Using the CISE proof tool [10], [11] the authors confirm that it is not sufficient for the replica that generates a move operation to check whether the operation introduces a cycle: like in Figure 2, two concurrent operations may be safe individually, but introduce a cycle when combined.…”

“…• We define a Conflict-free Replicated Data Type for trees that allow move operations without any coordination between replicas such as locking or consensus. As discussed in §2.3, this has previously been thought to be impossible to achieve [5], [6].…”

OpSets: Sequential Specifications for Replicated Datatypes (Extended Version)

“…Najafzadeh [40,41] also implemented a CRDT-based replicated filesystem, but chose a different approach: move operations must acquire a global lock before they can proceed, which ensures that conflicting concurrent move operations cannot occur in the first place. This conservative approach rules out move conflicts, but the resulting datatype is not strictly a CRDT, since some operations require strongly consistent synchronisation.…”

“…Najafzadeh [40,41] asserts that concurrent move operations on a tree cannot safely be implemented in a CRDT, since the precondition of a move operation is not stable. Najafzadeh suggests the use of locks to globally synchronise move operations, preventing a scenario such as that in Figure 4 from ever occurring.…”

“…This operation can be used to move a subtree to a new position within the tree, or to rename a key in a map, or to reorder items in a list. The OpSets approach enables a simple definition of this operation that had previously been thought impossible to implement without locking [40,41].…”