FlipIt: An LLVM Based Fault Injector for HPC

Abstract

Abstract. High performance computing (HPC) is increasingly subjected to faulty computations. The frequency of silent data corruptions (SDCs) in particular is expected to increase in emerging machines requiring HPC applications to handle SDCs. In this paper we, propose a robust fault injector structured through an LLVM compiler pass that allows simulation of SDCs in various applications. Although fault injection locations are enumerated at compile time, their activation is purely at runtime and based on a user-provided fault distribution. The robustness of our fault injector is in the ability to augment the runtime injection logic on a per application basis. This allows tighter control on the spacial, temporal, and probability of injected faults. The usability, scalability, and robustness of our fault injection is demonstrated with injecting faults into an algebraic multigird solver.

FlipTracker: Understanding Natural Error Resilience in HPC Applications

“…To capture and extract these patterns, however, a new method is required. While some methods exist to inject faults and statistically quantify their manifestation, such as random fault injection [2], [9], [10], [11], [12], and to use program analysis [13], [14], [15], [16], [17] to track errors on individual instructions, these methods miss the fine-grained information on error propagation as well as the context needed to explain, at a fine granularity, how errors propagate and consequently how natural resilient computations occur. In other words, these approaches do not provide the needed reasoning about how multiple computations work together to make an error disappear or to diminish its impact.…”

Refine

“…Recently, a number of compiler-based FI frameworks have been developed using the LLVM compiler [19]-some examples are LLFI [36], KULFI [32], VULFI [31], and FlipIt [6]. These methods are easy to use in large-scale parallel programs, and cooperate well with error propagation analysis frameworks since these frameworks typically operate at the compiler level too.…”

“…The way in which most LLVM IR-based FI tools inject faults into IR instructions is by adding a function call to the target instruction [6,31,32,36]. This function call performs large changes to the value of the result of the instruction or its arguments, and it may get inlined after optimization passes.…”

“…On the downside, though, compiler-based FI has three disadvantages compared to binary-level FI, all caused by the fact that existing methods perform injection at the compiler intermediate representation (IR) [6,31,32,36]: (1) not all low-level dynamic binary instructions are available at the IR level for FI; (2) instrumentation at the IR level interferes with code generation and optimizations-even if FI instrumentation is done after all IR optimizations are applied, the code that is input to the compiler backend can be significantly different from the original non-faulty code, which may generate very different (many times unoptimized) machine binary code; and (3) since code cannot be fully optimized (because of (2)), most frameworks incur significant (unnecessary) overhead, increasing the time to complete FI studies in large applications.…”

“…It is important to get an accurate picture of these proportions; for example, an application that experiences a large percentage of SDCs may require algorithmic error detection mechanisms, at the expense of runtime overhead. A concern in the HPC community is that a significant number of resilience studies have been based on this FI method [3,4,6,17,18,25,31,32,[34][35][36] (including our own work), which can potentially skew FI results and, in some cases, lead to incorrect conclusions. There has been research done in showing these inaccuracies.…”