Mixed Precision Block Fused Multiply-Add: Error Analysis and Application to GPU Tensor Cores

Double-precision floating-point arithmetic (FP64) has been the de facto standard for engineering and scientific simulations for several decades. Problem complexity and the sheer volume of data coming from various instruments and sensors motivate researchers to mix and match various approaches to optimize compute resources, including different levels of floating-point precision. In recent years, machine learning has motivated hardware support for half-precision floating-point arithmetic. A primary challenge in high-performance computing is to leverage reduced-precision and mixed-precision hardware. We show how the FP16/FP32 Tensor Cores on NVIDIA GPUs can be exploited to accelerate the solution of linear systems of equations Ax = b without sacrificing numerical stability. The techniques we employ include multiprecision LU factorization, the preconditioned generalized minimal residual algorithm (GMRES), and scaling and auto-adaptive rounding to avoid overflow. We also show how to efficiently handle systems with multiple right-hand sides. On the NVIDIA Quadro GV100 (Volta) GPU, we achieve a 4 × − 5 × performance increase and 5× better energy efficiency versus the standard FP64 implementation while maintaining an FP64 level of numerical stability.

show abstract

“…orders of magnitude smaller than the pure FP16 factorization [30], while retaining the high performance of the FP16 variant.…”

Section: (A) Auto-adaptive Roundingmentioning

confidence: 99%

Mixed-precision iterative refinement using tensor cores on GPUs to accelerate solution of linear systems

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Hardware that employs low-precision matrix multiplication with accumulation at high precision, such as the the NVIDIA tensor cores, requires a careful analysis that takes account of the internal precisions and the matrix size. A general such analysis, which quantifies the gain from the use of higher precision accumulation, is given in Blanchard et al (2020). A second question concerns the interaction of precision and dimension: an error bound with a constant nu (say) provides no information if nu>1, such as when n=104 and u is the unit roundoff for half precision (see Table 1).…”

Section: Dense Linear Algebramentioning

confidence: 99%

A survey of numerical linear algebra methods utilizing mixed-precision arithmetic

Abdelfattah

Anzt

Boman

et al. 2021

The International Journal of High Performance Computing Applica

Self Cite

108

View full text Add to dashboard Cite

The efficient utilization of mixed-precision numerical linear algebra algorithms can offer attractive acceleration to scientific computing applications. Especially with the hardware integration of low-precision special-function units designed for machine learning applications, the traditional numerical algorithms community urgently needs to reconsider the floating point formats used in the distinct operations to efficiently leverage the available compute power. In this work, we provide a comprehensive survey of mixed-precision numerical linear algebra routines, including the underlying concepts, theoretical background, and experimental results for both dense and sparse linear algebra problems.

show abstract

“…The main finding is that low-precision operations and usage of tensor cores increase the amount of correct data produced by the GPU, despite increasing the impact of numerical errors due to the use of lower-precision data. In order to quantify the accuracy of tensor cores, Blanchard et al (2020) provide a rounding error analysis of what they call a block fused multiply-add (FMA), a generalization of the multiply-accumulate operation in (1) in which the matrix sizes, the precisions of the arguments, and the internal precision of the accumulator are taken as parameters.…”

Section: Previous Workmentioning

confidence: 99%

Peer Review #2 of "Numerical behavior of NVIDIA tensor cores (v0.2)"

2021

View full text Add to dashboard Cite

We explore the floating-point arithmetic implemented in the NVIDIA tensor cores, which are hardware accelerators for mixed-precision matrix multiplication available on the Volta, Turing, and Ampere microarchitectures. Using Volta V100, Turing T4 and Ampere A100 graphics cards, we determine what precision is used for the intermediate results, whether subnormal numbers are supported, what rounding mode is used, in which order the operations underlying the matrix multiplication are performed, and whether partial sums are normalized. These aspects are not documented by NVIDIA, and we gain insight by running carefully designed numerical experiments on these hardware units. Knowing the answers to these questions is important if one wishes to: 1) accurately simulate NVIDIA tensor cores on conventional hardware; 2) understand the differences between results produced by code that utilizes tensor cores and code that uses only IEEE 754-compliant arithmetic operations; and 3) build custom hardware whose behavior matches that of NVIDIA tensor cores. As part of this work we provide a test suite that can be easily adapted to test newer versions of the NVIDIA tensor cores as well as similar accelerators from other vendors, as they become available. Moreover, we identify a non-monotonicity issue affecting floating point multi-operand adders if the intermediate results are not normalized after each step.

show abstract

Mixed Precision Block Fused Multiply-Add: Error Analysis and Application to GPU Tensor Cores

Cited by 41 publications

References 18 publications

Mixed-precision iterative refinement using tensor cores on GPUs to accelerate solution of linear systems

Mixed-precision iterative refinement using tensor cores on GPUs to accelerate solution of linear systems

A survey of numerical linear algebra methods utilizing mixed-precision arithmetic

Peer Review #2 of "Numerical behavior of NVIDIA tensor cores (v0.2)"

Contact Info

Product

Resources

About