Extreme modelling in practice

Abstract

Formal modelling is a powerful tool for developing complex systems. At MongoDB, we use TLA + to model and verify multiple aspects of several systems. Ensuring conformance between a specification and its implementation can add value to any specification; it can avoid transcription errors, prevent bugs as a large organization rapidly develops the specified code, and even keep multiple implementations of the same specification in sync. In this paper, we explore model-based testing as a tool for ensuring specification-implementation conformance. We attempted two case studies: model-based trace-checking (MBTC) in the MongoDB Server's replication protocol and model-based test-case generation (MBTCG) in MongoDB Realm Sync's operational transformation algorithm. We found MBTC to be impractical for testing that the Server conformed to a highly abstract specification. MBTCG was highly successful for Realm Sync, however. We analyze why one technique succeeded and the other failed, and advise future implementers making similar attempts at model-based testing.

Compositional Model Checking of Consensus Protocols Specified in TLA+ via Interaction-Preserving Abstraction

“…We will also investigate whether the IPA framework can be used to reduce the cost of code-level model checking of distributed system implementations. Given sufficient application of the IPA framework in realistic scenarios, we will investigate how to integrate the IPA framework into the extreme modeling [13] paradigm of distributed system design and implementation.…”

“…PolarFS is using TLA+ to precisely document the design of its ParallelRaft protocol, in order to effectively guarantee the reliability and maintainability of the protocol design and implementation [16]. MongoDB further leverages the formally specified design, verified by model checking, to conduct model-based test case generation and model-based trace checking on large scale system implementations [13].…”

Verifying Transactional Consistency of MongoDB

“…First, for simplicity, we have assumed that each procedure executes atomically. However, the implementation of MongoDB is highly concurrent [12], and needs to be modelled and verified more carefully. Second, MongoDB also supports non-transactional consistency, including tunable consistency [21] and causal consistency [26].…”

MET: Model Checking-Driven Explorative Testing of CRDT Designs and Implementations

“…We also propose an optimization concerning the model checking trace. We did not let the TLC model checker output the whole state transition graph like the work in [21]. Instead we let the CRDT models output only the final states.…”

“…MongoDB uses Model-Based Test Case Generation (MBTCG) to ensure the equivalence between the C++ and the Golang versions of the operational transformation (OT) implementations in MongoDB Realm Sync [21]. Met is inspired by the MBTCG technique of MongoDB.…”