CIFTS: A Coordinated Infrastructure for Fault-Tolerant Systems

Abstract

Considerable work has been done on providing fault tolerance capabilities for different software components on large-scale high-end computing systems. Thus far, however, these fault-tolerant components have worked insularly and independently and information about faults is rarely shared. Such lack of system-wide fault tolerance is emerging as one of the biggest problems on leadership-class systems. In this paper, we propose a coordinated infrastructure, named CIFTS, that enables system software components to share fault information with each other and adapt to faults in a holistic manner. Central to the CIFTS infrastructure is a Fault Tolerance Backplane (FTB) that enables fault notification and awareness throughout the software stack, including fault-aware libraries, middleware, and applications. We present details of the CIFTS infrastructure and the interface specification that has allowed various software programs, including MPICH2, MVAPICH, Open MPI, and PVFS, to plug into the CIFTS infrastructure. Further, through a detailed evaluation we demonstrate the nonintrusive low-overhead capability of CIFTS that lets applications run with minimal performance degradation.

Award ER25750: Coordinated Infrastructure for Fault Tolerance Systems Indiana University Final Report

“…In particular, we have published approximately 30 publications, presented 40 talks and 5 posters, and conducted Birds-of-a-Feather sessions and round-table discussion sessions every year for the past four years at the IEEE/ACM Supercomputing conference, and we have given more than 25 demonstrations of our research at various venues. The FTB design paper [36] has been cited two dozen times and by several independent sources, in less than two years. The community-targeted Birds-of-a-Feather sessions held at the IEEE/ACM Supercomputing conference [24][25][26][27] have been popular, with more than 50 attendees each year.…”

“…In summary, the majority of the FTB logic lies with the FTB agent. Further details about the design of the FTB implementation can be found in [36].…”

“…Details on CIFTS and the FTB software implementation have been a focus of several talks and presentations [28][29][30][31][32][33][34][35] given during this project.…”

A Case for Virtual Machine Based Fault Injection in a High-Performance Computing Environment

“…The STCI runtime provides basic monitoring and failure detectors that could be useful for experiment monitoring. The publish/subscribe services in STCI and CiFTS Fault Tolerant Backplane (FTB) [2] are also good candidates for implementing these event notification channels.…”

A Model-Based Case for Redundant Computation

“…We applaud the HPC community for increasing releases of failure data [2], and efforts to provide tools for measuring application efficiency [6,1,5], and recommend that simple tools be developed and made available for wide use. The required data resides within distinct communities (namely, administrators and users) -another example that solutions to the resilience issues facing exascale computing will require unified efforts to overcome [7,21]. With teamwork and open minds, the prospects of yet-more-extreme computing remains bright, despite the anticipated resilience challenges.…”