Lightweight Image Super-Resolution with Hierarchical and Differentiable Neural Architecture Search

Abstract

Single Image Super-Resolution (SISR) tasks have achieved significant performance with deep neural networks. However, the large number of parameters in CNNbased methods for SISR tasks require heavy computations. Although several efficient SISR models have been recently proposed, most are handcrafted and thus lack flexibility. In this work, we propose a novel differentiable Neural Architecture Search (NAS) approach on both the cell-level and network-level to search for lightweight SISR models. Specifically, the cell-level search space is designed based on an information distillation mechanism, focusing on the combinations of lightweight operations and aiming to build a more lightweight and accurate SR structure. The networklevel search space is designed to consider the feature connections among the cells and aims to find which information flow benefits the cell most to boost the performance. Unlike the existing Reinforcement Learning (RL) or Evolutionary Algorithm (EA) based NAS methods for SISR tasks, our search pipeline is fully differentiable, and the lightweight SISR models can be efficiently searched on both the cell-level and network-level jointly on a single GPU. Experiments show that our methods can achieve state-ofthe-art performance on the benchmark datasets in terms of PSNR, SSIM, and model complexity with merely 68G Multi-Adds for ×2 and 18G Multi-Adds for ×4 SR tasks. Code will be available at https://github.com/DawnHH/ DLSR-PyTorch.

Lightweight NAS Super-Resolution model

“…FERN [26] inserts group convolution into wide activation convolution [27,28] module to boost performance while maintaining fewer parameters and faster execution speed. DLSR [29] proposes the MRB structure,which is composed of a mixed layer, a residual connection, and a ReLU layer.The mixed layer is made up of multiple operations including normal convolution, separable convolution and dilated convolution [30]. PAN [31] employs the self-calibrated convolution scheme in their networks for efficient SR, which has fewest parameters among all participants in AIM 2020.…”

A Lightweight Image Super-Resolution Reconstruction Algorithm Based on the Residual Feature Distillation Mechanism

“…RFDN [ 36 ] applied residual feature distillation blocks, which are a variant of IMDB and are more powerful and flexible. DLSR [ 37 ] introduced a differentiable neural architecture search method to find more powerful fusion blocks based on RFDB.…”

NTIRE 2022 Challenge on Efficient Super-Resolution: Methods and Results

“…In detail, the shallow residual block in [51] is replaced by the proposed EDBB to construct the EFDB. Different from IMDN and RFDN utilizing global distillation connections to process input features progressively, neural architecture search (NAS) [30] is adopted to decide the feature connection paths. The searched structure is shown in the orange dashed box.…”