Cost-Efficient Super-Resolution Hardware Using Local Binary Pattern Classification and Linear Mapping for Real-Time 4K Conversion

Abstract

We propose a new hardware-friendly super-resolution (SR) algorithm using computationally simple feature extraction and regression methods, i.e., local binary pattern (LBP) and linear mapping, respectively. The proposed method pre-trains dedicated linear mapping kernels for different texture types of low-resolution (LR) image patches where the texture type is classified based on LBP features. On inference operation, a high-resolution (HR) image patch is reconstructed by multiplying an LR image patch with a linear mapping kernel, which is inferred by the LBP feature class of the corresponding LR patch. Since, the LBP is a highly efficient feature extraction operator for local texture classification, our method is extremely fast and power-efficient while showing competitive reconstruction quality to the latest machine learningbased SR techniques. We also present a fully pipe-lined hardware architecture and its implementation for real-time operations of the proposed SR method. The proposed SR algorithm has been implemented on a field-programmable-gate-array (FPGA) platform including Xilinx KCU105 that can process 63 frames-persecond (fps) while converting full-high-definition (FHD) images to 4K ultra-high-definition (UHD) images. Extensive experimental results show that the proposed proposed algorithm and its hardware implementation can achieve high reconstruction performance compared to the latest machine-learning-based SR methods while utilizing minimum hardware resources, thereby having remarkably less computational complexity. Sometimes, the latest deep-learning-based SR approaches offer slightly higher reconstruction quality, but they require significantly larger amount of hardware resources than the proposed method.

A Cost-Effective Interpolation for Multi-Magnification Super-Resolution

“…The Bicubic method is the conventional interpolation and is regarded as the comparative baseline of almost SR studies. Study SI [20], OLM-SI [21], HSI [22], and LBPC-SI [23], which are considered the SIbased SR methods, and the primary competitors by us. Two versions F and S of SR-LUT, also learning-based interpolation, are considered our competitors.…”

“…Since SI is introduced as a hardware-friendly method, the hardware implementation of SI (HSI) is provided in [22] as well, confirming that SI only costs very few hardware resources to achieve competitive performance. Based on [22], Bae et al [23] developed a more simplified classifier to achieve similar performance to [22]. Since the local binary pattern classification is used in [23], we call it LBPC-SI in this paper.…”

“…Based on [22], Bae et al [23] developed a more simplified classifier to achieve similar performance to [22]. Since the local binary pattern classification is used in [23], we call it LBPC-SI in this paper.…”