Assessment of Spatiotemporal Fusion Algorithms for Planet and Worldview Images

Kwan, Chiman; Zhu, Xiaolin; Gao, Feng; Chou, Bryan; Pérez, Daniel; Jiang, Li; Shen, Yuzhong; Koperski, Krzysztof; Marchisio, Giovanni

doi:10.3390/s18041051

Cited by 40 publications

(37 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Peak Signal-to-Noise Ratio (PSNR) [55] PNSR is related to Root Mean Squared Error (RMSE). The RMSE of two vectorized images S (ground truth) andŜ (prediction) is defined as…”

Section: Performance Metricsmentioning

confidence: 99%

Demosaicing of Bayer and CFA 2.0 Patterns for Low Lighting Images

Kwan

Larkin

2019

Electronics

Self Cite

View full text Add to dashboard Cite

It is commonly believed that having more white pixels in a color filter array (CFA) will help the demosaicing performance for images collected in low lighting conditions. However, to the best of our knowledge, a systematic study to demonstrate the above statement does not exist. We present a comparative study to systematically and thoroughly evaluate the performance of demosaicing for low lighting images using two CFAs: the standard Bayer pattern (aka CFA 1.0) and the Kodak CFA 2.0 (RGBW pattern with 50% white pixels). Using the clean Kodak dataset containing 12 images, we first emulated low lighting images by injecting Poisson noise at two signal-to-noise (SNR) levels: 10 dBs and 20 dBs. We then created CFA 1.0 and CFA 2.0 images for the noisy images. After that, we applied more than 15 conventional and deep learning based demosaicing algorithms to demosaic the CFA patterns. Using both objectives with five performance metrics and subjective visualization, we observe that having more white pixels indeed helps the demosaicing performance in low lighting conditions. This thorough comparative study is our first contribution. With denoising, we observed that the demosaicing performance of both CFAs has been improved by several dBs. This can be considered as our second contribution. Moreover, we noticed that denoising before demosaicing is more effective than denoising after demosaicing. Answering the question of where denoising should be applied is our third contribution. We also noticed that denoising plays a slightly more important role in 10 dBs signal-to-noise ratio (SNR) as compared to 20 dBs SNR. Some discussions on the following phenomena are also included: (1) why CFA 2.0 performed better than CFA 1.0; (2) why denoising was more effective before demosaicing than after demosaicing; and (3) why denoising helped more at low SNRs than at high SNRs.The baseline approach refers to a simple upsampling of the reduced resolution color image shown in Figure 2. The standard approach for CFA 2.0 is shown in Figure 2, which illustrates how to combine the interpolated luminance image with the reduced resolution color image to generate a full resolution color image. Electronics 2019, 8, 1444 4 of 56 combine the interpolated luminance image with the reduced resolution color image to generate a full resolution color image.Figure 2. Standard approach to demosaicing CFA 2.0 images. Image from [38].In the paper [16] written by us, we proposed a pansharpening approach to demosaicing CFA 2.0. This approach is illustrated in Figure 3. The missing pixels in the panchromatic band are interpolated. At the same time, the reduced resolution CFA is demosaiced. We then apply pansharpening to generate the full resolution color image. There are many pansharpening algorithms that can be used. Principal Component Analysis (PCA) [39], Smoothing Filter-based Intensity Modulation (SFIM) [40], Modulation Transfer Function Generalized Laplacian Pyramid (GLP) [41], MTF-GLP with High Pass Modulation (HPM) [42], Gram Schmidt (GS) [43], GS Adaptive (G...

show abstract

“…• Peak Signal-to-Noise Ratio (PSNR) [55] PNSR is related to Root Mean Squared Error (RMSE). The RMSE of two vectorized images S (ground truth) andŜ (prediction) is defined as…”

Section: Performance Metricsmentioning

confidence: 99%

Demosaicing of Bayer and CFA 2.0 Patterns for Low Lighting Images

Kwan

Larkin

2019

Electronics

Self Cite

View full text Add to dashboard Cite

show abstract

“…More details can be found in [148,149]. Another application scenario is the fusion of Worldview with Planet images [40]. A third temporal fusion study is for Landsat and Worldview images [150].…”

Section: Possibility Of Change Detection Using Temporally-fused Imagesmentioning

confidence: 99%

Methods and Challenges Using Multispectral and Hyperspectral Images for Practical Change Detection Applications

Kwan

2019

Information

Self Cite

View full text Add to dashboard Cite

Multispectral (MS) and hyperspectral (HS) images have been successfully and widely used in remote sensing applications such as target detection, change detection, and anomaly detection. In this paper, we aim at reviewing recent change detection papers and raising some challenges and opportunities in the field from a practitioner’s viewpoint using MS and HS images. For example, can we perform change detection using synthetic hyperspectral images? Can we use temporally-fused images to perform change detection? Some of these areas are ongoing and will require more research attention in the coming years. Moreover, in order to understand the context of our paper, some recent and representative algorithms in change detection using MS and HS images are included, and their advantages and disadvantages will be highlighted.

show abstract

“…Although there are many performance metrics in the literature, we selected the following ones: absolution difference (AD) 13 , root mean squared error (RMSE) 13 , peak signal-to-noise ratio (PSNR) 13 , cross correlation (CC) 13 , Erreur Relative Globale Adimensionnelle de Synthese (ERGAS) 13 , and structural similarity (SSIM) 13 .…”

Section: Performance Metricsmentioning

confidence: 99%

“…Another group focuses on integrating low spatial resolution high temporal resolution images with high spatial resolution low temporal resolution images. See papers [10][11][12][13] and references therein. Some applications include the fusion of MODIS and Landsat [10][11][12] , and the fusion of Planet and Worldview images 13 .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Fusion of landsat and worldview images

Kwan¹,

Chou

Yang

et al. 2019

Signal Processing, Sensor/Information Fusion, and Target Recognition XXVIII

Self Cite

View full text Add to dashboard Cite

Pansharpened Landsat images have 15 m spatial resolution with 16-day revisit periods. On the other hand, Worldview images have 0.5 m resolution after pansharpening but the revisit times are uncertain. We present some preliminary results for a challenging image fusion problem that fuses Landsat and Worldview (WV) images to yield a high temporal resolution image sequence at the same spatial resolution of WV images. Since the spatial resolution between Landsat and Worldview is 30 to 1, our preliminary results are mixed in that the objective performance metrics such as peak signal-to-noise ratio (PSNR), correlation coefficient (CC), etc. sometimes showed good fusion performance, but at other times showed poor results. This indicates that more fusion research is still needed in this niche application.

show abstract

Assessment of Spatiotemporal Fusion Algorithms for Planet and Worldview Images

Cited by 40 publications

References 19 publications

Demosaicing of Bayer and CFA 2.0 Patterns for Low Lighting Images

Demosaicing of Bayer and CFA 2.0 Patterns for Low Lighting Images

Methods and Challenges Using Multispectral and Hyperspectral Images for Practical Change Detection Applications

Fusion of landsat and worldview images

Contact Info

Product

Resources

About