PIRM Challenge on Perceptual Image Enhancement on Smartphones: Report

Abstract

This paper reviews the first challenge on efficient perceptual image enhancement with the focus on deploying deep learning models on smartphones. The challenge consisted of two tracks. In the first one, participants were solving the classical image super-resolution problem with a bicubic downscaling factor of 4. The second track was aimed at real-world photo enhancement, and the goal was to map low-quality photos from the iPhone 3GS device to the same photos captured with a DSLR camera. The target metric used in this challenge combined the runtime, PSNR scores and solutions' perceptual results measured in the user study. To ensure the efficiency of the submitted models, we additionally measured their runtime and memory requirements on Android smartphones. The proposed solutions significantly improved baseline results defining the state-of-the-art for image enhancement on smartphones. * A. Ignatov and R. Timofte ({andrey,radu.timofte}@vision.ee.ethz.ch, ETH Zurich) are the challenge organizers, while the other authors participated in the challenge. The Appendix contains the authors' teams and affiliations. PIRM 2018 Challenge webpage: http://ai-benchmark.org

MobiSR

“…As shown on Table 1, by constructing a shallower variant of RCAN, we are able to achieve comparable results with state-of-the-art SR models that are hand-optimized for increased efficiency. Notably, our reference model manages to outperform the winning model of the 2018 PIRM Challenge [25] on perceptual SR on mobile, FEQE, by 20% when run on the Hexagon DSP and achieves higher PSNR across all four SR datasets. Furthermore, m ref yields an average speedup of 16.01× (6.2× geo.…”

“…As a result, the number of multiply-add operations are typically counted in the billions as opposed to millions in discriminative networks [2]. Although the research community has made a few steps towards constructing efficient SR models that are optimized for mobile platforms, (1) running these models on-device is still costly and (2) the experiments presented in Section 4, the winning model [53] of the recent 2018 PIRM Challenge on perceptual SR on mobile [25] requires more than 1.4 s to ×4 upscale an image to 720p on the Hexagon DSP of Qualcomm Snapdragon 845.…”

Unsupervised Learning for Real-World Super-Resolution

“…While providing inferior PSNR compared to state-of-the-art, the super-resolved images experienced significantly better perceptual quality. Following this philosophy, the recent winner of the PIRM2018 [18] challenge ESRGAN [35], proposed further architectural improvements to further enhance the perceptual quality.…”

“…The GAN loss is multiplied by a weight λ, balancing the guidance of the two pixel-wise losses L VGG and L RaGAN against the GAN loss L GAN . Network architecture Our approach is agnostic to the specific architecture of the SR network S. For simplicity, we adopt the recently proposed ESRGAN architecture, which is the winner of the PIRM 2018 challenge [18]. It introduced a new building block called Residual-in-Residual Dense Blocks, improving stability of training.…”

DDet: Dual-Path Dynamic Enhancement Network for Real-World Image Super-Resolution

“…Owing to the rapid development of artificial intelligence technology, emerging applications, such as Alexa, Woogie, and Prisma, grow more prevalent than ever to change human lives. As a classical problem in computer vision, image super-resolution technology [1], [2], [3], [4], [5], [6], [7] also achieves tremendous progress and be widely used in many mobile devices, such as mobile phones, for photo enhancement [8]. With such a light-weight algorithm, mobile devices are capable of providing a high-quality photograph and free from purchase expensive sensors.…”