Investigating half precision arithmetic to accelerate dense linear system solvers

Abstract

The use of low-precision arithmetic in mixed-precision computing methods has been a powerful tool to accelerate numerous scientific computing applications. Artificial intelligence (AI) in particular has pushed this to current extremes, making use of half-precision floating-point arithmetic (FP16) in approaches based on neural networks. The appeal of FP16 is in the high performance that can be achieved using it on today's powerful manycore GPU accelerators, e.g., like the NVIDIA V100, that can provide 120 TeraFLOPS alone in FP16. We present an investigation showing that other HPC applications can harness this power too, and in particular, the general HPC problem of solving Ax = b, where A is a large dense matrix, and the solution is needed in FP32 or FP64 accuracy. Our approach is based on the mixed-precision iterative refinement technique-we generalize and extend prior advances into a framework, for which we develop architecture-specific algorithms and highly-tuned implementations that resolve the main computational challenges of efficiently parallelizing, scaling, and using FP16 arithmetic in the approach on high-end GPUs. Subsequently, we show for a first time how the use of FP16 arithmetic can significantly accelerate, as well as make more energy efficient, FP32 or FP64-precision Ax = b solvers. Our results are reproducible and the developments will be made available through the MAGMA library. We quantify in practice the performance, and limitations of the approach.

Accelerating the Solution of Linear Systems by Iterative Refinement in Three Precisions

“…A first step in this direction is the recent performance study of Haidar et al [16], which shows promising results.…”

Harnessing GPU Tensor Cores for Fast FP16 Arithmetic to Speed up Mixed-Precision Iterative Refinement Solvers

“…An investigation of similar iterative refinement methods on earlier generations of GPUs can be found in [11]. With the announcement of NVIDIA's V100 Tensor Cores.…”

“…Compared to our previous work in [11], the primary contribution of this paper is to propose and implement a high-performance framework for the mixed-precision iterative refinement solvers that makes use for the first time of GPU Tensor Core-accelerated FP16-TC. To this end, we will:…”

Accurate Sampling with Noisy Forces from Approximate Computing

“…While in many scientific applications the use of double-precision floating-point is most common, this precision is not always required. For example, iterative methods can exhibit resilience against low precision arithmetic as has been shown for the computation of inverse matrix roots [Lass et al 2018a] and for solving systems of linear equations [Angerer et al 2016;Haidar et al 2018Haidar et al , 2017Klavík et al 2014]. Mainly driven by the growing popularity of artificial neural networks [Gupta et al 2015], we can observe growing support of low-precision data types in hardware accelerators.…”