Proactive fault tolerance for HPC with Xen virtualization

Abstract

Large-scale parallel computing is relying increasingly on clusters with thousands of processors. At such large counts of compute nodes, faults are becoming common place. Current techniques to tolerate faults focus on reactive schemes to recover from faults and generally rely on a checkpoint/restart mechanism. Yet, in today's systems, node failures can often be anticipated by detecting a deteriorating health status.Instead of a reactive scheme for fault tolerance (FT), we are promoting a proactive one where processes automatically migrate from "unhealthy" nodes to healthy ones. Our approach relies on operating system virtualization techniques exemplified by but not limited to Xen. This paper contributes an automatic and transparent mechanism for proactive FT for arbitrary MPI applications. It leverages virtualization techniques combined with health monitoring and load-based migration. We exploit Xen's live migration mechanism for a guest operating system (OS) to migrate an MPI task from a health-deteriorating node to a healthy one without stopping the MPI task during most of the migration. Our proactive FT daemon orchestrates the tasks of health monitoring, load determination and initiation of guest OS migration. Experimental results demonstrate that live migration hides migration costs and limits the overhead to only a few seconds making it an attractive approach to realize FT in HPC systems. Overall, our enhancements make proactive FT a valuable asset for long-running MPI application that is complementary to reactive FT using full checkpoint/restart schemes since checkpoint frequencies can be reduced as fewer unanticipated failures are encountered. In the context of OS virtualization, we believe that this is the first comprehensive study of proactive fault tolerance where live migration is actually triggered by health monitoring.

Proactive process-level live migration and back migration in HPC environments

“…We next provide the performance comparison of our approach to another solution at the OS virtualization layer in the context of proactive FT of MPI applications [33]. The common benchmarks measured with both solutions on the same hardware were NPB BT, CG, LU and SP.…”

“…Furthermore, our approach provides a live migration mechanism, which supports continued execution of MPI applications during much of the migration time. This solution parallels live migration at the OS virtualization layer [10], which has been studied in the context of proactive FT of MPI applications [33], an approach that supports integrated health-based monitoring and proactive live migration over Xen guests. We contribute process-level live migration and demonstrate its superior efficiency to of OS-level virtualization.…”

Proactive Fault Tolerance Using Preemptive Migration

“…net) is a software effort to allow full access to IPMI information. OpenIPMI has already been used in Type 1 feedback-loop control solutions [6,13]. It can be used with any type of feedback-loop control.…”

“…Recent efforts in proactive FT primarily targeted two aspects, failure prediction [4,9] and process-or virtual-machine-level migration [13,6]. Other initial work focused on a proactive FT framework [12], which combines both to perform prediction triggered migration.…”

A Framework for Proactive Fault Tolerance

“…For instance [1] shows the interest of proactive fault tolerance for MPI applications, using Charm++ capabilities for the migration of MPI processes, migration and pause/unpause being two standard mechanisms for the implementation of proactive fault tolerance strategies. In [5] the authors show the benefit of using virtual machine (VM) migration for the implementation of proactive fault tolerance capabilities. However, these solutions directly implement a proactive fault tolerance policy into the system.…”