An Efficient Hardware Design for Accelerating Sparse CNNs With NAS-Based Models

Abstract

Deep convolutional neural networks (CNNs) have achieved remarkable performance at the cost of huge computation. As the CNN models become more complex and deeper, compressing CNNs to sparse by pruning the redundant connection in the networks has emerged as an attractive approach to reduce the amount of computation and memory requirement. On the other hand, FPGAs have been demonstrated to be an effective hardware platform to accelerate CNN inference. However, most existing FPGA accelerators focus on dense CNN models which are inefficient when executing sparse models as most of the arithmetic operations involve addition and multiplication with zero operands. In this work, we propose an accelerator with softwarehardware co-design for sparse CNNs on FPGAs. To efficiently deal with the irregular connections in the sparse convolutional layers, we propose a weight-oriented dataflow that exploits element-matrix multiplication as the key operation. Each weight is processed individually which yields low decoding overhead. Then we design an FPGA accelerator that features a tile lookup table (TLUT) and a channel multiplexer (CMUX). The tile look-up table is designed to match the index between sparse weights and input pixels. Using TLUT, the runtime decoding overhead is mitigated by using an efficient indexing operation. Moreover, we propose a weight layout to enable efficient on-chip memory access without conflicts. To cooperate with the weight layout, a channel multiplexer is inserted to locate the address. Last, we build a Neural Architecture Search (NAS) engine that leverages the reconfigurability of FPGAs to generate an efficient CNN model and choose the optimal hardware design parameters. Experiments demonstrate that our accelerator can achieve 223.4-309.0 GOP/s for the modern CNNs on Xilinx ZCU102, which provides a 2.4X-12.9X speedup over previous dense CNN accelerators on FPGAs. Our FPGA-aware NAS approach shows 2X speedup over MobileNetV2 with 1.5% accuracy loss.

Global to multi‐scale local architecture with hardwired CNN for 1‐ms tomato defect detection

“…This approach achieved an inference time in 19.42 ms for gesture classification. Li et al [38] proposed an architecture that skipped the calculation of similar data between frames in video applications, achieving 107.81 ms. Liang et al [39] used look-up tables to match the sparse weight and proposed compression format for memory access. MobileNetV2 achieved a delay of 27.3 ms using this architecture.…”

A Deep Learning Approach for Surface Crack Classification and Segmentation in Unmanned Aerial Vehicle Assisted Infrastructure Inspections

“…The accurate manual detection of cracks from the image/video data collected from a UAV can be tedious and time-consuming. Leveraging Artificial Intelligence (AI) and Machine Learning (ML) in combination with data captured through UAVs can significantly enhance the reliability and accuracy of inspections [ 13 ]. For this purpose, researchers have employed various image-detection methods, including morphological image processing [ 14 ], foreground–background separation [ 15 ], filtering [ 16 ], and percolation models [ 17 ].…”

Variable Precision Multiplier for CNN Accelerators Based on Booth Algorithm

“…Most edge devices do not process the training process that requires processing large amounts of data and only proceed with inference. Nevertheless, the CNN layer becomes gradually deeper, and studies to reduce the amount of memory access and computation, such as pruning, quantization, and compression, have also been actively conducted [11]- [14].…”