Computational Snapshot Multispectral Cameras: Toward dynamic capture of the spectral world

Cao, Xun; Yue, Tao; Lin, Xing; Lin, Stephen; Yuan, Xin; Dai, Qionghai; Carin, Lawrence; Brady, David J.

doi:10.1109/msp.2016.2582378

Cited by 244 publications

(124 citation statements)

References 47 publications

Supporting

Mentioning

124

Contrasting

Order By: Relevance

“…In pursuit of ultra-high spatial resolution, such as gigapixel, multi-camera systems and arrays have been developed [7], [8], [30]. In particular, cameras with different properties have been combined in a hybrid setup to sample and synthesize images from a variety of light components, such as hyperspectral imaging [11], low light imaging [27], and light fields [39], [43]. Pelican imaging and Light are two recent companies that launched products with multiple cameras on board that capture a diverse set of images and synthesize a desired image with high dynamic range, long-range depth, or light field [39] .…”

Section: Related Workmentioning

confidence: 99%

A Dual Camera System for High Spatiotemporal Resolution Video Acquisition

Asif

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

This paper presents a dual camera system for high spatiotemporal resolution (HSTR) video acquisition, where one camera shoots a video with high spatial resolution and low frame rate (HSR-LFR) and another one captures a low spatial resolution and high frame rate (LSR-HFR) video. Our main goal is to combine videos from LSR-HFR and HSR-LFR cameras to create an HSTR video. We propose an end-to-end learning framework, AWnet, mainly consisting of a FlowNet and a FusionNet that learn an adaptive weighting function in pixel domain to combine inputs in a frame recurrent fashion. To improve the reconstruction quality for cameras used in reality, we also introduce noise regularization under the same framework. Our method has demonstrated noticeable performance gains in terms of both objective PSNR measurement in simulation with different publicly available video and light-field datasets and subjective evaluation with real data captured by dual iPhone 7 and Grasshopper3 cameras. Ablation studies are further conducted to investigate and explore various aspects (such as noise regularization, camera parallax, exposure time, multiscale synthesis, etc) of our system to fully understand its capability for potential applications. respectively. His current research focuses on the next-generation video coding, energy-efficient communication, gigapixel streaming and deep learning. He is a co-recipient of 2018 ACM SIGCOMM Student Research Competition Finalist, 2018 PCM

show abstract

Section: Related Workmentioning

confidence: 99%

A Dual Camera System for High Spatiotemporal Resolution Video Acquisition

Asif

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…The values of the nonzero entries of the matrices are linked to Eqs. (18) and (19). Notice that one single sensor can measure several datacube elements.…”

Section: Matrix Forward Modelmentioning

confidence: 99%

Spectral Zooming and Resolution Limits of Spatial Spectral Compressive Spectral Imagers

Salazar

Parada-Mayorga

Arce

2019

IEEE Trans. Comput. Imaging

View full text Add to dashboard Cite

The recently introduced Spatial Spectral Compressive Spectral Imager (SSCSI) has been proposed as an alternative to carry out spatial and spectral coding using a binary onoff coded aperture. In SSCSI, the pixel pitch size of the coded aperture, as well as its location with respect to the detector array, play a critical role in the quality of image reconstruction. In this paper, a rigorous discretization model for this architecture is developed, based on a light propagation analysis across the imager. The attainable spatial and spectral resolution, and the various parameters affecting them, is derived through this process. Much like the displacement of zoom lens components leads to higher spatial resolution of a scene, a shift of the coded aperture in the SSCSI in reference to the detector leads to higher spectral resolution. This allows the recovery of spectrally detailed datacubes by physically displacing the mask towards the spectral plane. To prove the underlying concepts, computer simulations and experimental data are presented in this paper.

show abstract

“…An important aspect of hyperspectral imagers is the generation of vast amounts of data, accompanied by challenges in processing. Modern techniques, such as coded aperture snapshot spectral imaging (CASSI), are moving towards minimizing data collection at the acquisition stage by using computational methods to sample below the Nyquist-Shannon limit [30,31,32,33,34,35,36,37,38]. If a signal is sparse, with only few non-zero terms, a lot of redundant information is collected to meet the requirements of the sampling theorem.…”

Section: Fiber Imaging Spectrometermentioning

confidence: 99%

Snapshot fiber spectral imaging using speckle correlations and compressive sensing

French¹,

Gigan²,

Muskens³

2018

Opt. Express

View full text Add to dashboard Cite

Snapshot spectral imaging is rapidly gaining interest for remote sensing applications. Acquiring spatial and spectral data within one image promotes fast measurement times, and reduces the need for stabilized scanning imaging systems. Many current snapshot technologies, which rely on gratings or prisms to characterize wavelength information, are difficult to reduce in size for portable hyperspectral imaging. Here, we show that a multicore multimode fiber can be used as a compact spectral imager with sub-nanometer resolution, by encoding spectral information within a monochrome CMOS camera. We characterize wavelength-dependent speckle patterns for up to 3000 fiber cores over a broad wavelength range. A clustering algorithm is employed in combination with l1-minimization to limit data collection at the acquisition stage for the reconstruction of spectral images that are sparse in the wavelength domain. We also show that in the non-compressive regime these techniques are able to accurately reconstruct broadband information.

show abstract

Computational Snapshot Multispectral Cameras: Toward dynamic capture of the spectral world

Cited by 244 publications

References 47 publications

A Dual Camera System for High Spatiotemporal Resolution Video Acquisition

A Dual Camera System for High Spatiotemporal Resolution Video Acquisition

Spectral Zooming and Resolution Limits of Spatial Spectral Compressive Spectral Imagers

Snapshot fiber spectral imaging using speckle correlations and compressive sensing

Contact Info

Product

Resources

About