Compiler-Aware Neural Architecture Search for On-Mobile Real-time Super-Resolution

Wu, Yushu; Gong, Yifan; Zhao, Pu; Li, Yanyu; Zhan, Zheng; Niu, Wei; Tang, Hao; Qin, Minghai; Ren, Bin; Wang, Yanzhi

doi:10.1007/978-3-031-19800-7_6

Cited by 14 publications

(5 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…age restoration [92], and image/video super-resolution [93], [94]. Moreover, we follow OASIS [53] and use LPIPS [95] to evaluate the variation in the multi-model image synthesis on the ADE20K dataset.…”

Section: Resultsmentioning

confidence: 99%

Edge Guided GANs With Multi-Scale Contrastive Learning for Semantic Image Synthesis

Tang

Sun

Sebe

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

View full text Add to dashboard Cite

We propose a novel edge guided generative adversarial network with contrastive learning (ECGAN) for the challenging semantic image synthesis task. Although considerable improvements have been achieved by the community in the recent period, the quality of synthesized images is far from satisfactory due to three largely unresolved challenges. 1) The semantic labels do not provide detailed structural information, making it challenging to synthesize local details and structures; 2) The widely adopted CNN operations such as convolution, down-sampling, and normalization usually cause spatial resolution loss and thus cannot fully preserve the original semantic information, leading to semantically inconsistent results (e.g., missing small objects); 3) Existing semantic image synthesis methods focus on modeling "local" semantic information from a single input semantic layout. However, they ignore "global" semantic information of multiple input semantic layouts, i.e., semantic cross-relations between pixels across different input layouts. To tackle 1), we propose to use the edge as an intermediate representation which is further adopted to guide image generation via a proposed attention guided edge transfer module. Edge information is produced by a convolutional generator and introduces detailed structure information. To tackle 2), we design an effective module to selectively highlight class-dependent feature maps according to the original semantic layout to preserve the semantic information. To tackle 3), inspired by current methods in contrastive learning, we propose a novel contrastive learning method, which aims to enforce pixel embeddings belonging to the same semantic class to generate more similar image content than those from different classes. We further propose a novel multi-scale contrastive learning method that aims to push same-class features from different scales closer together being able to capture more semantic relations by explicitly exploring the structures of labeled pixels from multiple input semantic layouts from different scales. Experiments on three challenging datasets show that our methods achieve significantly better results than state-of-the-art approaches. The source code is available at https://github.com/Ha0Tang/ECGAN.

show abstract

Section: Resultsmentioning

confidence: 99%

Edge Guided GANs With Multi-Scale Contrastive Learning for Semantic Image Synthesis

Tang

Sun

Sebe

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, most are optimized for specific NPU mobile platforms, while the SR performance is insufficient. Wu et al [16] use a neural architecture search (NAS) framework with adaptive SR blocks to find an appropriate model to achieve real-time SR inference. However, it needs to retrain the model when the environment changes, which cannot be used on new devices directly.…”

Section: Cnn-based Efficient Srmentioning

confidence: 99%

“…SR-LUT [14] and SPLUT [15] can reconstruct images faster at the expense of severe performance degradation. Wu et al [16] explored a compiler-aware SR neural architecture search (NAS) framework to achieve real-time inference on GPU/DSP platforms for mobile devices. However, this work faces difficulties deploying or directly transferring pre-trained models to different hardware platforms with varying instruction architectures.…”

Section: Introductionmentioning

confidence: 99%

RepECN: Making ConvNets Better Again for Efficient Image Super-Resolution

Chen,

Qin,

Wen

2023

Sensors

View full text Add to dashboard Cite

Traditional Convolutional Neural Network (ConvNet, CNN)-based image super-resolution (SR) methods have lower computation costs, making them more friendly for real-world scenarios. However, they suffer from lower performance. On the contrary, Vision Transformer (ViT)-based SR methods have achieved impressive performance recently, but these methods often suffer from high computation costs and model storage overhead, making them hard to meet the requirements in practical application scenarios. In practical scenarios, an SR model should reconstruct an image with high quality and fast inference. To handle this issue, we propose a novel CNN-based Efficient Residual ConvNet enhanced with structural Re-parameterization (RepECN) for a better trade-off between performance and efficiency. A stage-to-block hierarchical architecture design paradigm inspired by ViT is utilized to keep the state-of-the-art performance, while the efficiency is ensured by abandoning the time-consuming Multi-Head Self-Attention (MHSA) and by re-designing the block-level modules based on CNN. Specifically, RepECN consists of three structural modules: a shallow feature extraction module, a deep feature extraction, and an image reconstruction module. The deep feature extraction module comprises multiple ConvNet Stages (CNS), each containing 6 Re-Parameterization ConvNet Blocks (RepCNB), a head layer, and a residual connection. The RepCNB utilizes larger kernel convolutions rather than MHSA to enhance the capability of learning long-range dependence. In the image reconstruction module, an upsampling module consisting of nearest-neighbor interpolation and pixel attention is deployed to reduce parameters and maintain reconstruction performance, while bicubic interpolation on another branch allows the backbone network to focus on learning high-frequency information. The extensive experimental results on multiple public benchmarks show that our RepECN can achieve 2.5∼5× faster inference than the state-of-the-art ViT-based SR model with better or competitive super-resolving performance, indicating that our RepECN can reconstruct high-quality images with fast inference.

show abstract

“…These works achieve real-time video reconstruction on a high-end GPU. In order to achieve real-time VSR on lower-end and mobile devices, experiments have been conducted on network pruning and neural architecture search [11,12,13,14].…”

Section: Introductionmentioning

confidence: 99%

Light-weight recurrent network for real-time video super-resolution

Wang¹,

Allebach²,

Qi³

2023

View full text Add to dashboard Cite

Real-time video super-resolution (VSR) has been considered a promising solution to improving video quality for video conferencing and media video playing, which requires low latency and short inference time. Although state-of-the-art VSR methods have been proposed with well-designed architectures, many of them are not feasible to be transformed into a real-time VSR model because of vast computation complexity and memory occupation. In this work, we propose a light-weight recurrent network for this task, where motion compensation offset is estimated by an optical flow estimation network, features extracted from the previous high-resolution output are aligned to the current target frame, and a hidden space is utilized to propagate long-term information. We show that the proposed method is efficient in real-time video super-resolution. We also carefully study the effectiveness of the existence of an optical flow estimation module in a lightweight recurrent VSR model and compare two ways of training the models. We further compare four different motion estimation networks that have been used in light-weight VSR approaches and demonstrate the importance of reducing information loss in motion estimation.

show abstract

Compiler-Aware Neural Architecture Search for On-Mobile Real-time Super-Resolution

Cited by 14 publications

References 65 publications

Edge Guided GANs With Multi-Scale Contrastive Learning for Semantic Image Synthesis

Edge Guided GANs With Multi-Scale Contrastive Learning for Semantic Image Synthesis

RepECN: Making ConvNets Better Again for Efficient Image Super-Resolution

Light-weight recurrent network for real-time video super-resolution

Contact Info

Product

Resources

About