An improved parallel singular value algorithm and its implementation for multicore hardware

Abstract

The enormous gap between the high-performance capabilities of today's CPUs and off-chip communication poses extreme challenges to the development of numerical software that is scalable and achieves high performance.In this article, we describe a successful methodology to address these challenges-starting with our algorithm design, through kernel optimization and tuning, and finishing with our programming model. All these lead to development of a scalable high-performance Singular Value Decomposition (SVD) solver. We developed a set of highly optimized kernels and combined them with advanced optimization techniques that feature fine-grain and cache-contained kernels, a task based approach, and hybrid execution and scheduling runtime, all of which significantly increase the performance of our SVD solver.Our results demonstrate a many-fold performance increase compared to currently available software. In particular, our software is two times faster than Intel's Math Kernel Library (MKL), a highly optimized implementation from the hardware vendor, when all the singular vectors are requested; it achieves a 5-fold speed-up when only 20% of the vectors are computed; and it is up to 10 times faster if only the singular values are required.

Neurostream: Scalable and Energy Efficient Deep Learning with Smart Memory Cubes

“…In [15] a rowmajor data layout has been proposed to improve DRAM's bandwidth efficiency and reduce bank conflicts in FPGA's BRAM banks. Also, tile-aware memory layouts have been previously proven effective for multi-core [36] and GPU implementations [37] of linear algebra algorithms, directly affecting their cache performance, bandwidth efficiency, and the degree of parallelism. In this paper, we introduce a general and flexible form called 4D-tiling (subsection IV-A) allowing for optimization of performance and energy efficiency under given constraints such as on-die SPM and DRAM bandwidth usage.…”

Look-ahead in the two-sided reduction to compact band forms for symmetric eigenvalue problems and the SVD

“…In the experiments, we employed square symmetric matrices for SEVP, and both square and rectangular matrices for the SVD, with random entries uniformly distributed in (0, 1), and dimensions of up to 10000 in steps of 500. We reiterate that the optimal bandwidth w depends not only on the implementation of the first stage, but also on that of the second stage, for which there exist multiple algorithms and tuned implementations, depending on the target architecture [9,18,19,10], the problem size, etc. For this reason, we decided to test the algorithms using six bandwidths: w = {32, 64, 96, 128, 192, 256}.…”

Porting the PLASMA Numerical Library to the OpenMP Standard

“…This is in contrast with LAPACK, where one tall panel (block of columns) is eliminated at a time, making it difficult to achieve cache efficiency and apply multithreading. In the course of the PLASMA project, tile algorithms have been developed for a wide range of algorithms, including: Cholesky, LU and QR factorizations [11,14,16], as well as reductions to band forms for solving the singular value problem or the eigenvalue problem [23,31].…”