A Deep Joint Network for Multispectral Demosaicking Based on Pseudo-Panchromatic Images

Liu, Shumin; Zhang, Yuge; Chen, Jie; Lim, Kwanghui; Rahardja, Susanto

doi:10.1109/jstsp.2022.3172865

Cited by 11 publications

(4 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Liu et al [ 25 ] proposed a new deep learning framework for multispectral demosaicing using pseudo-panchromatic images. The framework consists of two networks, the Deep PPI Generation Network (DPG-Net) and the Deep Demosaic Network (DDM-Net), which are used to generate and refine the PPI to improve image quality and recover high-frequency information in the demosaicing process.…”

Section: Related Work On Improving the Spatial Resolution Of Msfa Imagesmentioning

confidence: 99%

A Multi-Shot Approach for Spatial Resolution Improvement of Multispectral Images from an MSFA Sensor

Yao,

Ayikpa,

Gouton

et al. 2024

J. Imaging

View full text Add to dashboard Cite

Multispectral imaging technology has advanced significantly in recent years, allowing single-sensor cameras with multispectral filter arrays to be used in new scene acquisition applications. Our camera, developed as part of the European CAVIAR project, uses an eight-band MSFA to produce mosaic images that can be decomposed into eight sparse images. These sparse images contain only pixels with similar spectral properties and null pixels. A demosaicing process is then applied to obtain fully defined images. However, this process faces several challenges in rendering fine details, abrupt transitions, and textured regions due to the large number of null pixels in the sparse images. Therefore, we propose a sparse image composition method to overcome these challenges by reducing the number of null pixels in the sparse images. To achieve this, we increase the number of snapshots by simultaneously introducing a spatial displacement of the sensor by one to three pixels on the horizontal and/or vertical axes. The set of snapshots acquired provides a multitude of mosaics representing the same scene with a redistribution of pixels. The sparse images from the different mosaics are added together to get new composite sparse images in which the number of null pixels is reduced. A bilinear demosaicing approach is applied to the composite sparse images to obtain fully defined images. Experimental results on images projected onto the response of our MSFA filter show that our composition method significantly improves image spatial resolution and minimizes reconstruction errors while preserving spectral fidelity.

show abstract

Section: Related Work On Improving the Spatial Resolution Of Msfa Imagesmentioning

confidence: 99%

A Multi-Shot Approach for Spatial Resolution Improvement of Multispectral Images from an MSFA Sensor

Yao,

Ayikpa,

Gouton

et al. 2024

J. Imaging

View full text Add to dashboard Cite

show abstract

“…Inspired by the superior performance of data-driven methods in image restoration, some deep-learning-based approaches are proposed for computational spectral imaging, including black-box neural networks [23][24][25][26] and deep unfolding networks. [27][28][29] For example, Dijkstra, K. et al 23 used strided convolutions to softly rearrange the spectral mosaic pattern, which reduces a significant amount of computation.…”

Section: Introductionmentioning

confidence: 99%

“…Meng, Z. et al 24 presented a spatial-spectral self-attention module to model the spatial and spectral correlation with reasonable computation cost. Liu, S et al 25 proposed a novel deep CNN architecture based on pseudo-panchromatic images (PPI), combining the advantages of traditional PPI and deep-learning-based methods. Feng, K. et al 26 proposed a mosaic convolution-attention network that models joint spatial-spectral correlations.…”

Section: Introductionmentioning

confidence: 99%

TV-enhanced deep unfolding network for multispectral image demosaicing

Zhang,

Yang,

et al. 2024

Sixth Conference on Frontiers in Optical Imaging and Technology: Novel Imaging Systems

View full text Add to dashboard Cite

Multispectral image (MSI) contains a wealth of spatial information as well as spectral information, making it useful in the application of remote sensing, medical sciences, and beyond. However, traditional scanning-based imaging method is limited to low spatial or temporal resolution. Consequently, the reconstruction of high-resolution, clean, and complete MSI serves as an initial process for the numerous applications. This paper presents a novel deep unfolding network for demosaicing spectral mosaic images obtained through multispectral filter array (MSFA) imaging sensors. Concretely, the proposed network is unfolded from an iterative optimization process into an end-to-end training network, which can efficiently integrate the MSFA-based inherent degradation model with the powerful representation capability of deep neural networks. To further improve performance, a total-variation (TV) denoiser is plugged into the proposed network. Through end-to-end training, the hyperparameters within the optimization framework and TV denoiser are jointly optimized with the parameters of the neural network. Simulation results on CAVE and WHU-OHS datasets show that the proposed method outperforms state-of-the-art methods and improves the generalization capabilities to different MSFA settings.

show abstract

“…This method is suitable for any non-redundant 4 × 4 MSFA. In addition, a deep-learning-based multispectral-demosaicing method has been developed [16][17][18][19], which typically produces better results than traditional methods. However, deep-learning-based multispectral-demosaicing methods have a smaller dataset compared to deep-learningbased Bayer-filter-demosaicing methods.…”

Section: Introductionmentioning

confidence: 99%

Multispectral Demosaicing Based on Iterative-Linear-Regression Model for Estimating Pseudo-Panchromatic Image

Jeong,

Kim,

Kang

2024

Sensors

View full text Add to dashboard Cite

This paper proposes a method for demosaicing raw images captured by multispectral cameras. The proposed method estimates a pseudo-panchromatic image (PPI) via an iterative-linear-regression model and utilizes the estimated PPI for multispectral demosaicing. The PPI is estimated through horizontal and vertical guided filtering, with the subsampled multispectral-filter-array-(MSFA) image and low-pass-filtered MSFA as the guide image and filtering input, respectively. The number of iterations is automatically determined according to a predetermined criterion. Spectral differences between the estimated PPI and MSFA are calculated for each channel, and each spectral difference is interpolated using directional interpolation. The weights are calculated from the estimated PPI, and each interpolated spectral difference is combined using the weighted sum. The experimental results indicate that the proposed method outperforms the State-of-the-Art methods with regard to spatial and spectral fidelity for both synthetic and real-world images.

show abstract

A Deep Joint Network for Multispectral Demosaicking Based on Pseudo-Panchromatic Images

Cited by 11 publications

References 30 publications

A Multi-Shot Approach for Spatial Resolution Improvement of Multispectral Images from an MSFA Sensor

A Multi-Shot Approach for Spatial Resolution Improvement of Multispectral Images from an MSFA Sensor

TV-enhanced deep unfolding network for multispectral image demosaicing

Multispectral Demosaicing Based on Iterative-Linear-Regression Model for Estimating Pseudo-Panchromatic Image

Contact Info

Product

Resources

About