Design of Flexible Hardware Accelerators for Image Convolutions and Transposed Convolutions

Abstract

Nowadays, computer vision relies heavily on convolutional neural networks (CNNs) to perform complex and accurate tasks. Among them, super-resolution CNNs represent a meaningful example, due to the presence of both convolutional (CONV) and transposed convolutional (TCONV) layers. While the former exploit multiply-and-accumulate (MAC) operations to extract features of interest from incoming feature maps (fmaps), the latter perform MACs to tune the spatial resolution of the received fmaps properly. The ever-growing real-time and low-power requirements of modern computer vision applications represent a stimulus for the research community to investigate the deployment of CNNs on well-suited hardware platforms, such as field programmable gate arrays (FPGAs). FPGAs are widely recognized as valid candidates for trading off computational speed and power consumption, thanks to their flexibility and their capability to also deal with computationally intensive models. In order to reduce the number of operations to be performed, this paper presents a novel hardware-oriented algorithm able to efficiently accelerate both CONVs and TCONVs. The proposed strategy was validated by employing it within a reconfigurable hardware accelerator purposely designed to adapt itself to different operating modes set at run-time. When characterized using the Xilinx XC7K410T FPGA device, the proposed accelerator achieved a throughput of up to 2022.2 GOPS and, in comparison to state-of-the-art competitors, it reached an energy efficiency up to 2.3 times higher, without compromising the overall accuracy.

Design of a Low-Power Super-Resolution Architecture for Virtual Reality Wearable Devices

“…The proposed accelerator reaches an output throughput of ≈753 Mega pixels per second, corresponding to the generation of UHD images at a frame rate higher than 95 fps, which perfectly meets the target latency requirements [16]. When compared to prior works [20] and [23], which accelerate the final TCONV layer through innovative filter and pixel decomposition schemes that do not have impact on the computational complexity reduction, the proposed architecture sustains an output throughput 1.52 and 15.66 times higher, respectively, even processing larger input frames. At the same time, in comparison with the architecture proposed in [24], the amount of LUTs, FFs and BRAMs is reduced by 73.8%, 34.9% and 51.5%, respectively.…”

“…Furthermore, they introduce many unnecessary multiplications by zeros and require more complex strategies to access data memory, making video streaming unfeasible in some cases. Most of prior hardware designs, targeting either Field Programmable Gate Array (FPGA) [20], [23]- [24] or Application Specific Integrated Circuit (ASIC) [21]- [22], [25] technologies, rely on transforming the TCONV into multiple sub-convolutions. While these methods perform accurate TCONVs by re-arranging either the filter kernels [20], [21], [24] or the incoming pixels [23], they are not effective in reducing the number of involved MAC operations and introduce latency overheads and/or additional on-chip memory requirements.…”

“…Most of prior hardware designs, targeting either Field Programmable Gate Array (FPGA) [20], [23]- [24] or Application Specific Integrated Circuit (ASIC) [21]- [22], [25] technologies, rely on transforming the TCONV into multiple sub-convolutions. While these methods perform accurate TCONVs by re-arranging either the filter kernels [20], [21], [24] or the incoming pixels [23], they are not effective in reducing the number of involved MAC operations and introduce latency overheads and/or additional on-chip memory requirements. Differently, the ASIC design presented in [22] is based on slicing the low-resolution input image into multiple tiles that are processed by a decision network to establish the number of layers each tile has to pass through, according to its content.…”

“…the number of MAC operations) and the amount of memory needed to store the model parameters (Params) required by state-of-the-art competitors [10], [20]- [24]. Considering that [20] implements the model FSRCNN(25,5,1) and [21], [23]- [24] exploit the configuration FSRCNN(56,12,4), it can be observed that the proposed approach allows reducing computational and memory requirements by at least 5.11 and 1.09 times, respectively. In the following discussion, the PSNR/SSIM comparison is made considering the average of the percent deviations with reference to the available data.…”