An FPGA-based Convolution IP Core for Deep Neural Networks Acceleration

Abstract

The development of machine learning has madea revolution in various applications such as object detection,image/video recognition, and semantic segmentation. Neuralnetworks, a class of machine learning, play a crucial role inthis process because of their remarkable improvement overtraditional algorithms. However, neural networks are now goingdeeper and cost a significant amount of computation operations.Therefore they usually work ineffectively in edge devices thathave limited resources and low performance. In this paper, weresearch a solution to accelerate the neural network inferencephase using FPGA-based platforms. We analyze neural networkmodels, their mathematical operations, and the inference phasein various platforms. We also profile the characteristics thataffect the performance of neural network inference. Based on theanalysis, we propose an architecture to accelerate the convolutionoperation used in most neural networks and takes up most ofthe computations in networks in terms of parallelism, data reuse,and memory management. We conduct different experiments tovalidate the FPGA-based convolution core architecture as wellas to compare performance. Experimental results show that thecore is platform-independent. The core outperforms a quad-coreARM processor functioning at 1.2GHz and a 6-core Intel CPUwith speed-ups of up to 15.69 and 2.78, respectively.

A Review of the Optimal Design of Neural Networks Based on FPGA

“…As shown in Figure 6, according to different design concepts and requirements, FPGA-based neural network optimization technology can be roughly divided into optimization for data and operation, optimization for bandwidth, and optimization for memory and access, among others, which are introduced in detail below. [71][72][73][74][75][76][77][78], less computations [79][80][81], improve calculation speed [82][83][84][85], Winograd fast convolution algorithm [86][87][88][89][90][91], Im2col convolution optimization algorithm [92][93][94][95][96][97], pipelined design [98][99][100][101][102], Roofline model [103][104][105], ping-pong cache [106][107][108][109], input feature map reuse [110,111], filter reuse [111,112], convolutional reuse [110]…”

“…Aiming at the data itself, a method to reduce the data accuracy and computational complexity is usually used. For example, fixed-point Roof-line model [103][104][105], ping-pong cache [106][107][108][109], input feature map reuse [110,111], filter reuse [111,112], convolutional reuse [110][111][112], time reuse or space reuse [111], standardize data access and storage [113][114][115]).…”

“…In 2021, Hui Zhang et al [110] proposed an adaptive channel chunking strategy that reduced the size of on-chip memory and access to external memory, achieving better energy efficiency and utilizing multiplexing data and registering configuration in order to improve throughput and enhance accelerator versatility. In 2022, Xuan-Quang Nguyen et al [111] applied the data reuse technology that were based on time and space to the convolution operation in order to improve data utilization, reduce memory occupancy, reduce latency, and improve computational throughput. The same convolution calculation was about 2.78 times faster than the Intel Core i7-9750H CPU and 15.69 times faster than the ARM Cortex-A53.…”

“…Neural network optimization technology based on FPGA. (fixed-point quantization[71][72][73][74][75][76][77][78], less computations[79][80][81], improve calculation speed[82][83][84][85], Winograd fast convolution algorithm[86][87][88][89][90][91], Im2col convolution optimization algorithm[92][93][94][95][96][97], pipelined design[98][99][100][101][102], Roof-line model[103][104][105], ping-pong cache[106][107][108][109], input feature map reuse[110,111], filter reuse[111,112], convolutional reuse[110][111][112], time reuse or space reuse[111], standardize data access and storage…”