Accelerating Deterministic and Stochastic Binarized Neural Networks on FPGAs Using OpenCL

Abstract

Recent technological advances have proliferated the available computing power, memory, and speed of modern Central Processing Units (CPUs), Graphics Processing Units (GPUs), and Field Programmable Gate Arrays (FPGAs). Consequently, the performance and complexity of Artificial Neural Networks (ANNs) is burgeoning. While GPU-accelerated Deep Neural Networks (DNNs) currently offer state-of-the-art performance, they consume large amounts of power. Training such networks on CPUs is inefficient, as data throughput and parallel computation is limited. FPGAs are considered a suitable candidate for performance critical, low power systems, e.g. the Internet of Things (IOT) edge devices. Using the Xilinx SDAccel or Intel FPGA SDK for OpenCL development environment, networks described using the high-level OpenCL framework can be accelerated on heterogeneous platforms. Moreover, the resource utilization and power consumption of DNNs can be further enhanced by utilizing regularization techniques that binarize network weights. In this paper, we introduce, to the best of our knowledge, the first FPGA-accelerated stochastically binarized DNN implementations, and compare them to implementations accelerated on both GPUs and FPGAs. All our developed networks are trained and benchmarked using the popular MNIST and CIFAR-10 datasets. For our binarized and conventional FPGAbased networks, we achieve a >16-fold improvement in power consumption, compared to their GPU-accelerated counterparts. Also, our binarized FPGA-based networks require >25% shorter inference times, compared to their GPU-based counterparts.

Hardware Implementation of Deep Network Accelerators Towards Healthcare and Biomedical Applications

“…These tools allow engineers to describe their targeted hardware in high-level programming languages such as C to synthesize them to Register Transfer Level (RTL). The tools then offload the computationalcritical RTL to run as kernels on parallel processing platforms such as FPGAs [91].…”

Training Progressively Binarizing Deep Networks Using FPGAs

“…B INARIZATION has been used to augment the performance of Deep Neural Networks (DNNs), by quantizing network parameters to binary states, replacing many resourcehungry multiply-accumulate operations with simple accumulations [1]. It has been demonstrated that Binarized Neural Networks (BNNs) implemented on customized hardware can perform inference faster than conventional DNNs on stateof-the-art Graphics Processing Units (GPUs) [2], [3], while offering notable improvements in power consumption and resource utilizations [4]- [6]. However, there is still a performance gap between DNNs and conventional BNNs [7], which binarize parameters deterministically or stochastically.…”

Variation-aware Binarized Memristive Networks

“…Binarized Neural Networks (BNNs) [2], which perform binary MAC computations during forward and backward propagations, have demonstrated comparable performance to conventional DNNs, while significantly reducing resource and power utilizations [3]. On account of endurance concerns, ReRAM devices are ill-suited for implementing backward propagations, required during the training routine of BNNs where a large number of programming cycles are required.…”