Optimizing CNN-based Segmentation with Deeply Customized Convolutional and Deconvolutional Architectures on FPGA

Abstract

Convolutional Neural Networks (CNNs) based algorithms have been successful in solving image recognition problems, showing very large accuracy improvement. In recent years, deconvolution layers are widely used as key components in the state-of-the-art CNNs for end-to-end training and models to support tasks such as image segmentation and super resolution. However, the deconvolution algorithms are computationally intensive which limits their applicability to real time applications. Particularly, there has been little research on the efficient implementations of deconvolution algorithms on FPGA platforms which have been widely used to accelerate CNN algorithms by practitioners and researchers due to their high performance and power efficiency. In this work, we propose and develop deconvolution architecture for efficient FPGA implementation. FPGA-based accelerators are proposed for both deconvolution and CNN algorithms. Besides, memory sharing between the computation modules is proposed for the FPGA-based CNN accelerator as well as for other optimization techniques. A non-linear optimization model based on the performance model is introduced to efficiently explore the design space in order to achieve optimal processing speed of the system and improve power efficiency. Furthermore, a hardware mapping framework is developed to automatically generate the low-latency hardware design for any given CNN model on the target device. Finally, we implement our designs on Xilinx Zynq ZC706 board and the deconvolution accelerator achieves a performance of 90.1 GOPS under 200MHz working frequency and a performance density of 0.10 GOPS/DSP using 32-bit quantization, which significantly outperforms previous designs on FPGAs. A real-time application of scene segmentation on Cityscapes Dataset is used to evaluate our CNN accelerator on Zynq ZC706 board, and the system achieves a performance of 107 GOPS and 0.12 GOPS/DSP using 16-bit quantization, and supports up to 17 frames per second for 512x512 image inputs with a power consumption of only 9.6W.

Towards an Efficient Accelerator for DNN-Based Remote Sensing Image Segmentation on FPGAs

“…The popular DNN models for semantic segmentation include SegNet, U-Net, DeepLab, FCN, etc. These models are much more computationally intensive than the classification network models such as LeNet, AlexNet and GoogleNet [6].…”

“…The most relevant work to this paper is presented in [6], which optimized the computation of both Conv and Deconv layers for image segmentation. [6] proposed an efficient method to deal with the computational inefficiency occurred by deconvolution and used the design space exploration methodology to achieve the optimal resource allocation between Conv and Deconv modules. However, their design utilized separate modules for Conv and Deconv, which didn't share the DSPs for multipliers and led to the problem of under-utilization of resources.…”

“…Recently, Deep Neural Networks (DNNs) have become widely adopted in image segmentation problems and shown great accuracy improvement [6]. However, these DNN-based methods depend on very large computational complexity, in particular when processing RSIs with high spatial resolution.…”

“…FPGAs have shown very good performance for DNN-based segmentation acceleration [6], [11], and thus are considered promising to accelerate RSI segmentation methods in space platforms. The main challenge and limitations of previous designs of segmentation on FPGA are that: 1) unlike classification or detection networks such as VGG- 16 or AlexNet, segmentation networks consist of both convolutional (Conv) and deconvolutional (Deconv) layers which require different mathematical operations, making it difficult to design efficient architectures for them; 2) Deconv must insert zeros into the input image when implemented as Conv, leading to the sparsity of input as well as computational inefficiency due to multiplications with zeros [11].…”

Optimizing CNN-Based Hyperspectral Image Classification on FPGAs

“…Data Quantization. The main benefit of accelerating CNN models in FPGAs comes from the fact that CNNs are robust to low bitwidth quantization [11]. Instead of using the default double or single floating point precision in CPU, fixed-point precision can be used in FPGA-based CNN accelerator to achieve an efficient design optimized for performance and power efficiency [9,10].…”