High Performance Communication on Reconfigurable Clusters

Sheng, Jialian; Yang, Cheng‐Hong; Herbordt, Martin C.

doi:10.1109/fpl.2018.00044

Cited by 8 publications

(2 citation statements)

References 136 publications

(225 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The reason is that during IFP and OFP, the number of DSPs allocated to each layer is rounded to a multiple of K × K. With more FPGAs and more DSP resources, the effects of rounding error are lessened. [36,37,38].…”

Section: Cluster-level Performance Evaluationmentioning

confidence: 99%

See 1 more Smart Citation

A Framework for Acceleration of CNN Training on Deeply-Pipelined FPGA Clusters with Work and Weight Load Balancing

Geng

Wang

Sanaullah

et al. 2018

2018 28th International Conference on Field Programmable Logic and Applications (FPL)

Self Cite

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) have revolutionized numerous applications, but the demand for ever more performance remains unabated. Scaling DNN computations to larger clusters is generally done by distributing tasks in batch mode using methods such as distributed synchronous SGD. Among the issues with this approach is that to make the distributed cluster work with high utilization, the workload distributed to each node must be large, which implies nontrivial growth in the SGD mini-batch size. In this paper, we propose a framework called FPDeep, which uses a hybrid of model and layer parallelism to configure distributed reconfigurable clusters to train DNNs. This approach has numerous benefits. First, the design does not suffer from batch size growth. Second, novel workload and weight partitioning leads to balanced loads of both among nodes. And third, the entire system is a fine-grained pipeline. This leads to high parallelism and utilization and also minimizes the time features need to be cached while waiting for back-propagation. As a result, storage demand is reduced to the point where only on-chip memory is used for the convolution layers. We evaluate FPDeep with the Alexnet, VGG-16, and VGG-19 benchmarks. Experimental results show that FPDeep has good scalability to a large number of FPGAs, with the limiting factor being the FPGA-to-FPGA bandwidth. With 6 transceivers per FPGA, FPDeep shows linearity up to 83 FPGAs. Energy efficiency is evaluated with respect to GOPs/J. FPDeep provides, on average, 6.36x higher energy efficiency than comparable GPU servers.Deep convolutional neural networks (CNNs) have revolutionized applications such as image classification and object recognition [1,2,3,4]. As there remains an open-ended demand for more complex networks and larger datasets, new computing solutions are critical.Distributed synchronous stochastic gradient descent (SGD) has enabled large-scale CNN training by partitioning SGD mini-batches into smaller data batches that can then be processed in parallel and so accelerate CNN training [5]. A drawback of this method is scalability: to enable continued high utilization as the number of nodes increases, each node must be allocated an ever larger workload. But larger mini-batches slow training convergence. Thus, while larger clusters provide increased computation capacity, the training time is not proportionally reduced [5].

show abstract

Section: Cluster-level Performance Evaluationmentioning

confidence: 99%

“…We use a cycle-accurate simulator to evaluate cluster-level performance. As each transceiver (of that generation) can reach a maximum rate of 28 Gb/s, using 6 transceivers per FPGA achieves this number [36,37,38].…”

Section: Cluster-level Performance Evaluationmentioning

confidence: 99%

A Framework for Acceleration of CNN Training on Deeply-Pipelined FPGA Clusters with Work and Weight Load Balancing

Geng

Wang

Sanaullah

et al. 2018

2018 28th International Conference on Field Programmable Logic and Applications (FPL)

Self Cite

View full text Add to dashboard Cite

show abstract

Reconfigurable switches for high performance and flexible MPI collectives

Haghi

Guo

Xiong

et al. 2021

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

There has been much effort in offloading MPI collective operations into hardware.But while NIC-based collective acceleration is well-studied, offloading their processing into the switching fabric, despite numerous advantages, has been much more limited.A major problem with fixed logic implementations is that either only a fraction of the possible collective communication is accelerated or that logic is wasted in the applications that do not need a particular capability. Using reconfigurable logic has numerous advantages: exactly the required operations can be implemented; the level of desired performance can be specified; and new, possibly complex, operations can be defined and implemented. We have designed an in-switch collective accelerator, MPI-FPGA, and demonstrated its use with seven MPI collectives and over a set of benchmarks and proxy applications (MiniApps). The accelerator uses a novel two-level switch design containing fully pipelined vectorized aggregation logic units. Essential to this work is providing support for sub-communicator collectives that enables communicators of arbitrary shape, and that is scalable to large systems. A streaming interface improves the performance for long messages. While this reconfigurable design is generally applicable, we prototype it with an FPGA-centric cluster. A sample MPI-FPGA design in a direct network achieves considerable speedups over conventional clusters in the most likely scenarios. We also present results for indirect networks with reconfigurable high-radix switches and show that this approach is competitive with SHArP technology for the subset of operations that SHArP supports. MPI-FPGA is fully integrated into MPICH and is transparent to MPI applications.

show abstract