HPC Programming on Intel Many-Integrated-Core Hardware with MAGMA Port to Xeon Phi

Dongarra, Jack; Gates, Mark; Haidar, Azzam; Jia, Yulu; Kabir, K. Habibul; Łuszczek, Piotr; Tomov, Stanimire

doi:10.1155/2015/502593

Cited by 17 publications

(7 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, we can't accelerate neural networks without dedicated devices/libraries [28], [29]. The sparse networks are presented by CSR [30] or CSC and calculated by cuSparse [31] or MKL [32].…”

Section: A Network Pruningmentioning

confidence: 99%

Roulette: A Pruning Framework to Train a Sparse Neural Network From Scratch

et al. 2021

View full text Add to dashboard Cite

Due to space and inference time restrictions, finding an efficient and sparse sub-network from a dense and over-parameterized network is critical for deploying neural networks on edge devices. Recent efforts explore obtaining a sparse sub-network by performing network pruning during training procedures to reduce training costs, such as memory and floating-point operations (FLOPs). However, these works take more than 1.4× the total number of iterations and try all possible pruning parameters manually to obtain sparse sub-networks. In this paper, we present a pruning framework Roulette to train a sparse network from scratch. First, we propose a novel method to train a sparse network by Pruning through the lens of Loss Landscape iteratively and automatically (PLL). We do a theoretical analysis that the curvature of the loss function is higher in the initial phase and can conduct us to start network pruning. According to our results on CIFAR-10/100 and ImageNet dataset, PLL saves up to 4× training FLOPs than prior works while maintaining comparable or even better accuracy. Then we design push and pull operations to synchronize the pruned weights on different GPUs during training, scaling PLL to multiple GPUs linearly. To our knowledge, Roulette is the first network pruning framework supporting multiple GPUs linearly. INDEX TERMS Network pruning, inference acceleration, model compression, multiple GPUs Model FLOPs GPU hours ResNet50 4

show abstract

Section: A Network Pruningmentioning

confidence: 99%

Roulette: A Pruning Framework to Train a Sparse Neural Network From Scratch

et al. 2021

View full text Add to dashboard Cite

show abstract

“…As for libraries, some works, for specific kind of applications, address the portability problem using native programming models. For example, MAGMA library [6] provides a unified programming environment for heterogeneous systems using both CPUs and accelerators, such as a GPU or Intel Xeon Phi, for dense linear algebra algorithms. However, most of heterogeneous libraries rely on the OpenCL abstraction.…”

Section: Introductionmentioning

confidence: 99%

Supporting the Xeon Phi Coprocessor in a Heterogeneous Programming Model

Moreton-Fernandez

Rodriguez-Gutiez

González-Escribano

et al. 2017

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Supercomputers are becoming more heterogeneous. They are composed by several machines with different computation capabilities and different kinds and families of accelerators, such as GPUs or Intel Xeon Phi coprocessors. Programming these machines is a hard task, that requires a deep study of the architectural details, for exploiting efficiently each computational unit. In this paper, we present an extension of a heterogeneous programming model, in order to also support Intel Xeon Phi coprocessors. This contribution extends an existing heterogeneous programming library, by taking advantage of both the GPU communication model and the CPU execution model of the original library. Our experimental results show that using our approach, the programming effort needed for changing the target devices is highly reduced, by for example reusing the 97% of the code between a GPU implementation and a Xeon Phi implementation for the Mandelbrot benchmark.

show abstract

“…Some specific-domain libraries address the portability problem internally using several native programming models. For example, MAGMA library [45] provides a unified programming environment for heterogeneous systems using both CPUs and accelerators, such as GPUs or Intel Xeon Phi, for dense linear algebra algorithms, with complete different implementations for each type of device that cannot be easily used together.…”

Section: Proposals For Standardizing Parallel Programming | 37mentioning

confidence: 99%

Easing parallel programming on heterogeneous systems

Fernández¹

View full text Add to dashboard Cite

The use of parallel computing systems frequently represents the only scalable way to solve HPC (High performance Computing) problems in reasonable execution times. The current trend in high performance computing platforms is to include in the same machine several parallel devices, of different type and architectures, and to interconnect them to form highly parallel and heterogeneous distributed systems. Programming efficient and portable parallel applications that can really exploit these systems, imposes specific and complex challenges to the programmers. A programmer must be proficient in distributed-memory communication tools or layers, shared-memory programming models, and specific programming models for the available co-processors, in order to create hybrid programs that will exploit all the machine capabilities. Moreover, she also has to deal with the proper workload distribution among the different nodes and devices, assigning to each one an amount of workload related to their computation power and features. Nowadays, all these issues should be solved by the programmer, making the programming of heterogeneous platforms an actual challenge. This PhD. Thesis addresses several main problems related to the parallel programming for highly heterogeneous and distributed systems. It first tackles problems to allow the developing of efficient coordination codes, portable across different kind of devices, accelerators, and architectures. Then, it also targets problems related to the data communication and partition issues concerning the use of devices in distributed-memory systems. In this dissertation we introduce abstractions, mechanisms, and methods to solve many of these problems. We also discuss their practical application to develop research prototypes and actual programming tools. Experimental works conducted using these tools validates the applicability of the proposed techniques and the portability, efficiency, and versatility of the programs that can be obtained.

show abstract

HPC Programming on Intel Many-Integrated-Core Hardware with MAGMA Port to Xeon Phi

Cited by 17 publications

References 17 publications

Roulette: A Pruning Framework to Train a Sparse Neural Network From Scratch

Roulette: A Pruning Framework to Train a Sparse Neural Network From Scratch

Supporting the Xeon Phi Coprocessor in a Heterogeneous Programming Model

Easing parallel programming on heterogeneous systems

Contact Info

Product

Resources

About