Soft error resilient QR factorization for hybrid system with GPGPU

Abstract

As the general purpose graphics processing units (GPGPU) are increasingly deployed for scientific computing for its raw performance advantages compared to CPUs, the fault tolerance issue has started to become more of a concern than before when they were exclusively used for graphics applications. The pairing of GPUs with CPUs to form a hybrid computing systems for better flexibility and performance creates a massive amounts of computations that have a higher possibility to be affected by transient error -a soft error that silently modifies data causing errors to pass unnoticed. This is despite the fact that the newest Fermi generation of GPUs from NVIDIA are equipped with error correcting units to protect their memories. This problem is particularly serious for applications that employ numerical linear algebra since large sections of data are often modified between steps, and therefore even a single error could eventually propagate into a large area of result. In order to give protection to dense linear algebra computations on such hybrid systems, we developed an algorithm that is resilient to soft errors. We chose the right-looking Householder QR factorization as a demonstration of our algorithm for a hybrid system that features both GPUs and CPUs. Algorithm based fault tolerance (ABFT) is used to protect from errors in the trailing matrix and the right factor, while a checkpointing method is used to ensure the left factor is error-free. This work is based on a previous study of fault tolerance in matrix factorizations. Our contribution includes (1) a stable multiple-error checkpointing and recovery mechanism for the left-factor, which is also scalable in performance in the hybrid execution environment and does not cause severe performance degradation. (2) optimized Givens rotation utilities on the GPU to efficiently reduce an upper Hessenberg matrix to upper triangular form, and (3) a recovery algorithm based on QR update inside a hybrid system. Experimental results show that, our fault tolerant QR factorization can successfully detect and correct data altered by soft errors in both the left and right factors and we observe a decreasing percentage of overhead as the matrix size grows.

Combating Alarm Fatigue: The Quest for More Accurate and Safer Clinical Monitoring Equipment

“…This improvement in device specificity would result in significantly fewer false alarms and therefore reduce AF. At the same time, the challenges of "unpredictable code" and "interrupted or corrupt data" have been noted and may represent an important safety issue due to the potential for missing data or data misinterpretation, especially when using memory-intensive applications on devices that are continually operating for prolonged periods of time [89,[92][93][94][95].…”

Mitigating Silent Data Corruptions in Integer Matrix Products: Toward Reliable Multimedia Computing on Unreliable Hardware

“…Existing techniques that can ensure reliability to SDCs comprise two categories: (i) algorithm-based fault tolerance 1 (ABFT)-i.e., methods using checksums specifically tailored to the algorithm under consideration-that can reliably detect (and possibly correct) up to a limited number of SDCs [13], [17], [19], [25], [39], [46], [47], [60]; (ii) systems with dual modular redundancy (DMR), where all non-coinciding SDCs can be detected if the same operation is duplicated in two separate processors (or threads) that cross-validate their results [21], but SDCs cannot be corrected without using triple modular redundancy (TMR) [23].…”

“…The "best-case" SDC distribution for this implementation would be having x SDCs spread in such a way that only one SDC occurs in a row or column of a partition. Since mABFT requires one division operation for error location and one subtraction operation for error correction [19], [54] , 2x operations would be required in order to correct the x detected SDCs.…”

A backward/forward recovery approach for the preconditioned conjugate gradient method

“…Since we are dealing with sparse matrices, we expect n ′ to be very small, and hence the computation of the norm to be accurate. Moreover, since the right hand side in (14) does not depend on x, it can be computed just once for a given matrix and weight vector. Clearly, using (14) as tolerance parameter guarantees no false positive (a computation without any error that is considered as faulty), but allows false negatives (an iteration during which an error occurs without being detected) when the perturbations of the result are small.…”

“…The pioneering paper of Huang and Abraham [10] describes an algorithm capable of detecting and correcting a single silent error striking a dense matrix-matrix multiplication by means of row and column checksums. ABFT protection has been successfully applied to dense LU [11], LU with partial pivoting [12], Cholesky [13] and QR [14] factorizations, and more recently to sparse kernels like SpMxV (matrix-vector product) and triangular solve [15]. The overhead induced by ABFT is usually small, which makes it a good candidate for error detection at each iteration of the CG algorithm.…”