Skip-Concatenated Image Super-Resolution Network for Mobile Devices

Abstract

Single-image super-resolution technology has been widely studied in various applications to improve the quality and resolution of degraded images acquired from noise-sensitive low-resolution sensors. As most studies on single-image super-resolution focused on the development of deep learning networks operating on high-performance GPUs, this study proposed an efficient and lightweight super-resolution network that enables real-time performance on mobile devices. To replace the relatively slow element-wise addition layer on mobile devices, we introduced a skip connection layer by directly concatenating a lowresolution input image with an intermediate feature map. In addition, we introduced weighted clipping to reduce the quantization errors commonly encountered during float-to-int8 model conversion. Moreover, a reparameterization method was selectively applied without increasing the cost in terms of inference time and number of parameters. Based on the contributions, the proposed network has been recognized as the best solution in Mobile AI & AIM 2022 Real-Time Single-Image Super-Resolution Challenge with PSNR of 30.03 dB and NPU runtime of 19.20 ms.

Baojia Sun, Gongdao Zhang v Shanghai Huili Group Co, Ltd, Shanghai Huili Flooring Products Co, Ltd

“…Our method achieves the best scores on every benchmark dataset. AsConvSR takes only 3.91 ms, which is close to the bicubic interpolation, and achieves an improvement on RLFN [26] FMEN [16] RTSRN [18] AsConvSR-L HR Bicubic IMDN [22] RFDN [33] RLFN [26] FMEN [16] RTSRN [18] AsConvSR-L PSNR for more than 1dB. Our AsConvSR is the only method that achieves real-time performance with 1080p inputs.…”

“…In this section, we compare AsConvSR with the stateof-the-art efficient super-resolution models. Specifically, IMDN [22], RFDN [33], RLFN [26], FMAN [16], and RTSRN [18] are chosen for experiments.…”

“…Processing time and memory bandwidth of a neural network are highly related to the resolution of input images. RTSRN [18] can achieve real-time performance for a 360p input on a Tesla V100 GPU. However, runtime of this model would increase to 37.91 ms (see Section 4.2) when the input resolution is 1080p.…”

“…Previous works like RFDN [33] and RTSRN [18] have proposed efficient networks for super-resolution, but they mainly aim at lower input resolutions like 540p and 640p. Their real-time (30fps, below 33ms per image) performance cannot be guaranteed on higher input resolutions.…”

Feature-Domain Adaptive Contrastive Distillation for Efficient Single Image Super-Resolution

“…The practical applicability of complex SISR models is limited in resource-constrained devices, such as mobile or IoT devices; thus, efficient and lightweight SISR models are required. To satisfy this demand, lightweight SISR models with a better trade-off between efficiency and performance quality have been proposed [16], [17], [19]- [25]. Among the aforementioned methods, knowledge distillation (KD) [26]based approaches exhibit the following distinctive advantages: 1) KD promotes the inheritance of the knowledge of large teacher networks and improves performance without modifying the existing network structure at industrial sites, and 2) KD can be combined with pruning and network design methods by including additional loss terms to achieve greater performance improvement [27], [28].…”