On recent results in demosaicing of Samsung 108MP CMOS sensor using deep learning

Kim, Irina; Lim, Dongpan; Seo, Young–Il; Lee, Jeongguk; Choi, Young-Dae; Song, Seongwook

doi:10.1109/tensymp52854.2021.9550945

Cited by 6 publications

(4 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For Nona CFA (3 × 3), Sugawara et al [5] developed a GAN based on a spatial-asymmetric attention module to minimize artifacts. Kim et al [6] demonstrated that a duplex pyramid network (DPN) [23] achieves low visual artifacts and effective edge restoration in demosaicing images captured by a SAMSUNG CMOS sensor.…”

Section: Related Work a Deep Learning Based Demosaicingmentioning

confidence: 99%

“…Image signal processors (ISPs) then convert these RAW images into high-quality RGB images through processes such as demosaicing, denoising, white balancing, and gamma correction. Although deep learning-based ISP techniques have greatly improved image reconstruction quality [1], [2], [3], [4], most existing methods focus on standard Bayer CFAs (Figure 1a), with only a few considering non-Bayer CFAs [5], [6]. With non-Bayer CFAs causing different image statistics compared to standard Bayer filtered images, ISPs should be re-optimized, particularly for the latest Q×Q Bayer CFAs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image Sensors

Cho

Lee

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Deep learning-based image signal processor (ISP) models for mobile cameras can generate high-quality images that rival those of professional DSLR cameras. However, their computational demands often make them unsuitable for mobile settings. Additionally, modern mobile cameras employ non-Bayer color filter arrays (CFA) such as Quad Bayer, Nona Bayer, and Q×Q Bayer to enhance image quality, yet most existing deep learning-based ISP (or demosaicing) models focus primarily on standard Bayer CFAs. In this study, we present PyNET-Q×Q, a lightweight demosaicing model specifically designed for Q×Q Bayer CFA patterns, which is derived from the original PyNET. We also propose a knowledge distillation method called progressive distillation to train the reduced network more effectively. Consequently, PyNET-Q×Q contains less than 2.5% of the parameters of the original PyNET while preserving its performance. Experiments using Q×Q images captured by a prototype Q×Q camera sensor show that PyNET-Q×Q outperforms existing conventional algorithms in terms of texture and edge reconstruction, despite its significantly reduced parameter count. Code and partial datasets can be found at https://github.com/Minhyeok01/PyNET-QxQ.INDEX TERMS Bayer filter, color filter array (CFA), demosaicing, image signal processor (ISP), knowledge distillation, non-Bayer CFA, Q×Q Bayer CFA.

show abstract

Section: Related Work a Deep Learning Based Demosaicingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image Sensors

Cho

Lee

et al. 2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Deep learning approaches to demosaicing has been applied [25], [26], [27], [28]. Previously, many researches focused on the bayer CFA demosaicing, but there are researches on Quad bayer pattern and Nona pattern demosaicing also [29], [30]. Deep learning methods have better image quality in complex CFA pattern demosaicing although they require high computation cost.…”

Section: Releated Workmentioning

confidence: 99%

On quantization of convolutional neural networks for image restoration

Youngil¹,

Irina²,

Jeongguk³

et al. 2022

Self Cite

View full text Add to dashboard Cite

Recently, commercial vision sensors hit the mobile market. To achieve that, computer vision networks had to be quantized. However, this topic was not studied well for Image Signal processor (ISP) challenging image restoration tasks, being crucially important for hardware implementation, as well as for deployment on hardware accelerators, e.g. Neural Processors Units (NPU).In this paper, we studied the effect of the quantization of deep learning network on image quality. We tried various quantization on raw RGBW image demosaicing. Experimental results show that 10 bit weight quantization can sustain image quality at the similar level with floating-point network. 8 bit quantized network shows slight degradation in objective image quality and mild visual artifacts.Although network weight's bit-depth can be significantly reduced for computer vision tasks, our finding shows that it is not true for raw image restoration tasks: at least 10 bit weights are required to provide sufficient image quality. However, we can save some memory on feature maps bit-depth. We can conclude that network bit depth is critical for raw image restoration.

show abstract

“…Various demosaicing approaches are studied, including color difference interpolation [17], edge directional interpolation [18], frequency domain filtering [8], [4], [5], and reconstruction methods [19], [20]. However, when it comes to rarely used patterns, such as Quad Bayer, Nonacell or RGBW, there are only a few works addressing demosaicing of those CFAs [2][12][13] [14].…”

Section: Contributionsmentioning

confidence: 99%

DePhaseNet: A deep convolutional network using phase differentiated layers and frequency based custom loss for RGBW image sensor demosaicing

Irina¹,

Youngil²,

Dongpan³

et al. 2022

Self Cite

View full text Add to dashboard Cite

Panchromatic Color Filter Arrays with white signal were introduced a while ago, e. g. RGBW Color Filter Array (CFA), assuming to have better resolution in lowlight due to panchromatic signal. However, there is no successful RGBW image sensor in the industry targeting mobile cameras until now. In this work, we introduce a novel Samsung RGBW image sensor and we study its performance in a popular remosaic scenario. We propose a DePhaseNet -a deep fully convolutional network to solve RGBW remosaicing or demosaicing problem. We propose to have 3 layers of phase differentiated inputs and custom frequency-based loss function for each layer. Proposed method successfully suppresses False Colors inherent to RGBW sensor due to heavily undersampled colors. By using this method, we were able not only to increase details preservation, but also increased color reproduction. We found that RGBW sensor is beneficial not only in low light scenarios, but also in widely spread remosaic scenarios. Experiments show improvement in image quality, yielding CPSNR of 42dB for Kodak dataset, reaching the bar of Bayer CFA demosaicing result. Proposed method advances state-of-the-art in RGBW demosaic by 6dB in CPSNR.

show abstract

On recent results in demosaicing of Samsung 108MP CMOS sensor using deep learning

Cited by 6 publications

References 4 publications

PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image Sensors

PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image Sensors

On quantization of convolutional neural networks for image restoration

DePhaseNet: A deep convolutional network using phase differentiated layers and frequency based custom loss for RGBW image sensor demosaicing

Contact Info

Product

Resources

About