Mixed precision algorithms in numerical linear algebra

Abstract

Today’s floating-point arithmetic landscape is broader than ever. While scientific computing has traditionally used single precision and double precision floating-point arithmetics, half precision is increasingly available in hardware and quadruple precision is supported in software. Lower precision arithmetic brings increased speed and reduced communication and energy costs, but it produces results of correspondingly low accuracy. Higher precisions are more expensive but can potentially provide great benefits, even if used sparingly. A variety of mixed precision algorithms have been developed that combine the superior performance of lower precisions with the better accuracy of higher precisions. Some of these algorithms aim to provide results of the same quality as algorithms running in a fixed precision but at a much lower cost; others use a little higher precision to improve the accuracy of an algorithm. This survey treats a broad range of mixed precision algorithms in numerical linear algebra, both direct and iterative, for problems including matrix multiplication, matrix factorization, linear systems, least squares, eigenvalue decomposition and singular value decomposition. We identify key algorithmic ideas, such as iterative refinement, adapting the precision to the data, and exploiting mixed precision block fused multiply–add operations. We also describe the possible performance benefits and explain what is known about the numerical stability of the algorithms. This survey should be useful to a wide community of researchers and practitioners who wish to develop or benefit from mixed precision numerical linear algebra algorithms.

Responsibly Reckless Matrix Algorithms for HPC Scientific Applications

“…A more adequate approach is to prescribe the FP precision on a tile-bytile basis, referred to as responsible in Figure 2. This tile-centric approach relies on comparing the Frobenius norm of a tile against the Frobenius norm of the overall matrix, as explained in [6]. Depending on the ratio of norms and the required numerical accuracy by the application, a tilecentric decision is made on the FP precision before the matrix operations proceed.…”

“…Finally, can also leverage tile low-rank (TLR) approximations in addition to MP tile algorithms to further reduce the memory footprint and time-to-solution, leading to a reckless but responsible algorithm that can still ensure adequate accuracy [6]. With the help of PaRSEC dynamic runtime system [16], the Cholesky factorization relies on a hybrid data distribution that mitigates the load imbalance between tasks next to and far from the diagonal with high/low algorithmic complexity, respectively.…”

“…Moreover, by using a tile-centric responsible MP variant, the ToR computation can capture the needed accuracy per tile [6] and ensure numerical robustness without requiring a priori knowledge on the data sparsity structure of the covariance matrix. Combining MP with tile low-rank matrix approximations (i.e., reckless but responsible MP+TLR algorithms) is an interesting avenue to further satisfy real-time requirements and will be investigated in future work.…”

“…Unfortunately, this may sometimes lead into a reckless usage of MP algorithms, without proper numerical validation, especially in situations where multidisciplinary collaboration may not be fostered. Recent work [6] provides mathematical insight and error analyses that will eventually engender a wider adoption from domain scientists.…”

A Mixed-Precision Implementation of the Density Matrix Renormalization Group

“…Because of the potential advantages, using a relatively lower precision is a common approach for performance optimization in machine learning 25 and high-performance computing 26 and has also been applied in various scientific areas such as numerical linear algebra, [27][28][29][30][31] molecular dynamics, 32 and lattice quantum chromodynamics. 33 In quantum chemistry, some mixed-precision (MP) electron repulsion integral evaluation methods 34,35 have been implemented.…”