A Novel Utilization of Image Registration Techniques to Process Mastcam Images in Mars Rover With Applications to Image Fusion, Pixel Clustering, and Anomaly Detection

Ayhan, Bulent; Dao, Minh N.; Kwan, Chiman; Chen, Hua-Mei; Bell, James F.; Kidd, Richard

doi:10.1109/jstars.2017.2716923

Cited by 73 publications

(43 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As shown in Figure 1a, for each 2 × 2 block, there are two green pixels, one red pixel, and one blue pixel. Even in the Mastcam onboard the Mars rover Curiosity [2][3][4][5], the Bayer pattern has been used for the RGB images. The main reason for using the Bayer pattern is to reduce cost.…”

Section: Introductionmentioning

confidence: 99%

Demosaicing of Bayer and CFA 2.0 Patterns for Low Lighting Images

Kwan

Larkin

2019

Electronics

Self Cite

View full text Add to dashboard Cite

It is commonly believed that having more white pixels in a color filter array (CFA) will help the demosaicing performance for images collected in low lighting conditions. However, to the best of our knowledge, a systematic study to demonstrate the above statement does not exist. We present a comparative study to systematically and thoroughly evaluate the performance of demosaicing for low lighting images using two CFAs: the standard Bayer pattern (aka CFA 1.0) and the Kodak CFA 2.0 (RGBW pattern with 50% white pixels). Using the clean Kodak dataset containing 12 images, we first emulated low lighting images by injecting Poisson noise at two signal-to-noise (SNR) levels: 10 dBs and 20 dBs. We then created CFA 1.0 and CFA 2.0 images for the noisy images. After that, we applied more than 15 conventional and deep learning based demosaicing algorithms to demosaic the CFA patterns. Using both objectives with five performance metrics and subjective visualization, we observe that having more white pixels indeed helps the demosaicing performance in low lighting conditions. This thorough comparative study is our first contribution. With denoising, we observed that the demosaicing performance of both CFAs has been improved by several dBs. This can be considered as our second contribution. Moreover, we noticed that denoising before demosaicing is more effective than denoising after demosaicing. Answering the question of where denoising should be applied is our third contribution. We also noticed that denoising plays a slightly more important role in 10 dBs signal-to-noise ratio (SNR) as compared to 20 dBs SNR. Some discussions on the following phenomena are also included: (1) why CFA 2.0 performed better than CFA 1.0; (2) why denoising was more effective before demosaicing than after demosaicing; and (3) why denoising helped more at low SNRs than at high SNRs.The baseline approach refers to a simple upsampling of the reduced resolution color image shown in Figure 2. The standard approach for CFA 2.0 is shown in Figure 2, which illustrates how to combine the interpolated luminance image with the reduced resolution color image to generate a full resolution color image. Electronics 2019, 8, 1444 4 of 56 combine the interpolated luminance image with the reduced resolution color image to generate a full resolution color image.Figure 2. Standard approach to demosaicing CFA 2.0 images. Image from [38].In the paper [16] written by us, we proposed a pansharpening approach to demosaicing CFA 2.0. This approach is illustrated in Figure 3. The missing pixels in the panchromatic band are interpolated. At the same time, the reduced resolution CFA is demosaiced. We then apply pansharpening to generate the full resolution color image. There are many pansharpening algorithms that can be used. Principal Component Analysis (PCA) [39], Smoothing Filter-based Intensity Modulation (SFIM) [40], Modulation Transfer Function Generalized Laplacian Pyramid (GLP) [41], MTF-GLP with High Pass Modulation (HPM) [42], Gram Schmidt (GS) [43], GS Adaptive (G...

show abstract

Section: Introductionmentioning

confidence: 99%

Demosaicing of Bayer and CFA 2.0 Patterns for Low Lighting Images

Kwan

Larkin

2019

Electronics

Self Cite

View full text Add to dashboard Cite

show abstract

“…Moreover, our goal is to raise some challenges and practical issues in change detection using MS and HS images from a practitioner's point of view. In the past five years, we have been working on applications such as border monitoring [12,17,27,28], refugee camp growth monitoring [29], Mars exploration using multispectral imagers [30][31][32][33][34], airplane detection using satellite images [29], etc. From those applications, we observed some challenging issues that, in our opinion, have been overlooked by academics, and which are still unresolved.…”

Section: Introductionmentioning

confidence: 99%

Methods and Challenges Using Multispectral and Hyperspectral Images for Practical Change Detection Applications

Kwan

2019

Information

Self Cite

View full text Add to dashboard Cite

Multispectral (MS) and hyperspectral (HS) images have been successfully and widely used in remote sensing applications such as target detection, change detection, and anomaly detection. In this paper, we aim at reviewing recent change detection papers and raising some challenges and opportunities in the field from a practitioner’s viewpoint using MS and HS images. For example, can we perform change detection using synthetic hyperspectral images? Can we use temporally-fused images to perform change detection? Some of these areas are ongoing and will require more research attention in the coming years. Moreover, in order to understand the context of our paper, some recent and representative algorithms in change detection using MS and HS images are included, and their advantages and disadvantages will be highlighted.

show abstract

“…The Mars rover, Curiosity, has many instruments onboard for Mars data collection and in-situ surface characterization [1]. Alpha Particle X-ray Spectrometer (APXS) [2], Laser Induced Breakdown Spectrometer (LIBS) [3] [4], and Mastcam [5]- [8] are well-known ones. Quite a few of these instruments are imagers that fight for limited bandwidth to transmit data back to Earth.…”

Section: Introductionmentioning

confidence: 99%

Compression Algorithm Selection for Multispectral Mastcam Images

Kwan¹,

Larkin²,

Budavari³

et al. 2019

SIPIJ

Self Cite

View full text Add to dashboard Cite

The two mast cameras (Mastcam) onboard the Mars rover, Curiosity, are multispectral imagers with nine bands in each camera. Currently, the images are compressed losslessly using JPEG, which can achieve only two to three times compression. We present a two-step approach to compressing multispectral Mastcam images. First, we propose to apply principal component analysis (PCA) to compress the nine bands into three or six bands. This step optimally compresses the 9-band images through spectral correlation between the bands. Second, several well-known image compression codecs, such as JPEG, JPEG-2000 (J2K), X264, and X265, in the literature are applied to compress the 3-band or 6-band images coming out of PCA. The performance of different algorithms was assessed using four well-known performance metrics. Extensive experiments using actual Mastcam images have been performed to demonstrate the proposed framework. We observed that perceptually lossless compression can be achieved at a 10:1 compression ratio. In particular, the performance gain of an approach using a combination of PCA and X265 is at least 5 dBs in terms peak signal-to-noise ratio (PSNR) at a 10:1 compression ratio over that of JPEG when using our proposed approach.

show abstract

A Novel Utilization of Image Registration Techniques to Process Mastcam Images in Mars Rover With Applications to Image Fusion, Pixel Clustering, and Anomaly Detection

Cited by 73 publications

References 23 publications

Demosaicing of Bayer and CFA 2.0 Patterns for Low Lighting Images

Demosaicing of Bayer and CFA 2.0 Patterns for Low Lighting Images

Methods and Challenges Using Multispectral and Hyperspectral Images for Practical Change Detection Applications

Compression Algorithm Selection for Multispectral Mastcam Images

Contact Info

Product

Resources

About