A scalable approach to solving dense linear algebra problems on hybrid CPU‐GPU systems

Abstract

SUMMARYAiming to fully exploit the computing power of all CPUs and all GPUs on hybrid CPU-GPU systems to solve dense linear algebra problems, we design a class of heterogeneous tile algorithms to maximize the degree of parallelism, to minimize the communication volume, as well as to accommodate the heterogeneity between CPUs and GPUs. The new heterogeneous tile algorithms are executed upon our decentralized dynamic scheduling runtime system, which schedules a task graph dynamically and transfers data between compute nodes automatically. The runtime system uses a new distributed task-assignment protocol to solve data dependencies between tasks without any coordination between processing units. By overlapping computation and communication through dynamic scheduling, we are able to attain scalable performance for the double-precision Cholesky factorization and QR factorization. Our approach demonstrates performance better than both vendor (e.g., Intel MKL) and open source libraries (e.g., StarPU, PLASMA) in the following four possible environments: heterogeneous clusters with GPUs, conventional clusters without GPUs, sharedsystems with multiple GPUs, and shared-memory multicore computers.

Evaluating Accuracy and Efficiency of HPC Solvers for Sparse Linear Systems with Applications to PDEs

“…The scientific community has a great interest in the numerical solution of linear systems, as confirmed by a long-standing tradition in software development [17], [10], [36].…”

New iterative methods for dense linear systems

“…where A is nonsingular, x is unknown and b is known and nonzero. The case where A is dense can be solved numerically by direct methods, iterative methods and parallel methods [11].…”