Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates

Abstract

Oobleck enables resilient distributed training of large DNN models with guaranteed fault tolerance. It takes a planningexecution co-design approach, where it first generates a set of heterogeneous pipeline templates and instantiates at least 𝑓 + 1 logically equivalent pipeline replicas to tolerate any 𝑓 simultaneous failures. During execution, it relies on alreadyreplicated model states across the replicas to provide fast recovery. Oobleck provably guarantees that some combination of the initially created pipeline templates can be used to cover all available resources after 𝑓 or fewer simultaneous failures, thereby avoiding resource idling at all times. Evaluation on large DNN models with billions of parameters shows that Oobleck provides consistently high throughput, and it outperforms state-of-the-art fault tolerance solutions like Bamboo and Varuna by up to 13.9×. CCS Concepts:• Computer systems organization → Dependable and fault-tolerant systems and networks; Distributed architectures; Neural networks.