Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI &amp; AIM 2022 Challenge: Report

Ignatov, Andrey; Timofte, Radu; Liu, Shuai; Feng, Chaoyu; Bai, Furui; Wang, Xiaotao; Lei, Lei; Yi, Ziyao; Xiang, Yan; Liu, Zibin; Li, Shaoqing; Shi, Keming; Kong, Dehui; Xu, Ke; Kwon, Minsu; Wu, Yaqi; Zheng, Jiesi; Fan, Zhihao; Wu, Xun; Zhang, Feng; No, Albert; Cho, Minhyeok; Chen, Zewen; Zhang, Xiaze; Li, Ran; Wang, Juan; Wang, Zhiming; Conde, Marcos V.; Choi, Ui-Jin; Ershov, Egor; Hui, Zheng; Dong, Mengchuan; Lou, Xin; Zhou, Wei; Pang, Cong; Qin, Haina; Cai, Mingxuan

doi:10.1007/978-3-031-25066-8_3

Cited by 6 publications

(2 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Alongside image quality, runtime performance is also a crucial consideration on smartphone devices. Therefore, subsequent challenges [15], [17] placed more stringent constraints on on-device performance while providing a unified platform to evaluate proposed models on the target device. Similar trends were observed in these works regarding loss functions and a monolithic architecture, but notably, most methods could obtain close to real-time performance on the device.…”

Section: Related Workmentioning

confidence: 99%

Advocating Pixel-Level Authentication of Camera-Captured Images

Punnappurath,

Zhao,

Abdelhamed

et al. 2024

IEEE Access

View full text Add to dashboard Cite

The authenticity of digital images posted online and shared on social media is often questioned due to the ability of photo-editing software to alter image content and generative AI methods that can produce visually compelling deepfakes. Only images directly produced by cameras are deemed unaltered and beyond suspicion, as they have not undergone any modifications. However, there is a recent trend among camera manufacturers to integrate AI-based modules into the dedicated onboard hardware, specifically the image signal processor (ISP), responsible for processing the captured sensor image into the final saved image for users. Many of these AI modules utilize perceptual or generative losses during training, which can ''hallucinate'' image content. While this hallucinated content often manifests as small details and textures, there are instances where these regions unintentionally impact the interpretation of the entire image. This paper aims to bring attention to this issue and advocate for in-camera strategies to validate the authenticity of camera-captured images at a pixel level. We propose the creation of an "authenticity" mask that could be stored as additional metadata with each image. This information can be extracted and overlaid on the image to easily identify the hallucinated regions. Considering the widespread implications of image authenticity (e.g., in courtroom evidence, news broadcasts, and other media forms), we anticipate that authentication metadata will become a standard practice for any ISP utilizing AI.

show abstract

Section: Related Workmentioning

confidence: 99%

Advocating Pixel-Level Authentication of Camera-Captured Images

Punnappurath,

Zhao,

Abdelhamed

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Hardware Image Signal Processors (ISPs) are low-level image processing hardwares that convert RAW data to RGB images. Due to their high reliability and high efficiency, they are widely used in many fields, e.g., camera phones (Ignatov et al 2023;Ratnasingam 2019) and video surveillance (Lee et al 2015;Xu et al 2018;Baina and Dublet 1995). Generally, a typical hardware ISP pipeline consists of a set of serialized processing blocks (e.g., denoising block and sharpening block), each of which contains multiple parameters, resulting in a large number of parameters, which affects the quality of the generated RGB image (Bardenet et al 2013;Yahiaoui et al 2019;Yogatama and Mann 2014).…”

Section: Introductionmentioning

confidence: 99%

RL-SeqISP: Reinforcement Learning-Based Sequential Optimization for Image Signal Processing

Sun,

Zhao,

Wei

et al. 2024

AAAI

View full text Add to dashboard Cite

Hardware image signal processing (ISP), aiming at converting RAW inputs to RGB images, consists of a series of processing blocks, each with multiple parameters. Traditionally, ISP parameters are manually tuned in isolation by imaging experts according to application-specific quality and performance metrics, which is time-consuming and biased towards human perception due to complex interaction with the output image. Since the relationship between any single parameter’s variation and the output performance metric is a complex, non-linear function, optimizing such a large number of ISP parameters is challenging. To address this challenge, we propose a novel Sequential ISP parameter optimization model, called the RL-SeqISP model, which utilizes deep reinforcement learning to jointly optimize all ISP parameters for a variety of imaging applications. Concretely, inspired by the sequential tuning process of human experts, the proposed model can progressively enhance image quality by seamlessly integrating information from both the image feature space and the parameter space. Furthermore, a dynamic parameter optimization module is introduced to avoid ISP parameters getting stuck into local optima, which is able to more effectively guarantee the optimal parameters resulting from the sequential learning strategy. These merits of the RL-SeqISP model as well as its high efficiency are substantiated by comprehensive experiments on a wide range of downstream tasks, including two visual analysis tasks (instance segmentation and object detection), and image quality assessment (IQA), as compared with representative methods both quantitatively and qualitatively. In particular, even using only 10% of the training data, our model outperforms other SOTA methods by an average of 7% mAP on two visual analysis tasks.

show abstract

Residual Feature Distillation Channel Spatial Attention Network for ISP on Smartphone

Zheng

Fan

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report

Cited by 6 publications

References 55 publications

Advocating Pixel-Level Authentication of Camera-Captured Images

Advocating Pixel-Level Authentication of Camera-Captured Images

RL-SeqISP: Reinforcement Learning-Based Sequential Optimization for Image Signal Processing

Residual Feature Distillation Channel Spatial Attention Network for ISP on Smartphone

Contact Info

Product

Resources

About