From compressive sampling to compressive tasking: retrieving semantics in compressed domain with low bandwidth

Abstract

High-throughput imaging is highly desirable in intelligent analysis of computer vision tasks. In conventional design, throughput is limited by the separation between physical image capture and digital post processing. Computational imaging increases throughput by mixing analog and digital processing through the image capture pipeline. Yet, recent advances of computational imaging focus on the “compressive sampling”, this precludes the wide applications in practical tasks. This paper presents a systematic analysis of the next step for computational imaging built on snapshot compressive imaging (SCI) and semantic computer vision (SCV) tasks, which have independently emerged over the past decade as basic computational imaging platforms. SCI is a physical layer process that maximizes information capacity per sample while minimizing system size, power and cost. SCV is an abstraction layer process that analyzes image data as objects and features, rather than simple pixel maps. In current practice, SCI and SCV are independent and sequential. This concatenated pipeline results in the following problems: i) a large amount of resources are spent on task-irrelevant computation and transmission, ii) the sampling and design efficiency of SCI is attenuated, and iii) the final performance of SCV is limited by the reconstruction errors of SCI. Bearing these concerns in mind, this paper takes one step further aiming to bridge the gap between SCI and SCV to take full advantage of both approaches. After reviewing the current status of SCI, we propose a novel joint framework by conducting SCV on raw measurements captured by SCI to select the region of interest, and then perform reconstruction on these regions to speed up processing time. We use our recently built SCI prototype to verify the framework. Preliminary results are presented and the prospects for a joint SCI and SCV regime are discussed. By conducting computer vision tasks in the compressed domain, we envision that a new era of snapshot compressive imaging with limited end-to-end bandwidth is coming.

Hybrid Multi-Dimensional Attention U-Net for Hyperspectral Snapshot Compressive Imaging Reconstruction

“…In addition to spectral SCI reconstruction as shown in this work, we do believe our network can be used in medical images [ 59 ], image compression [ 60 ], temporal compressive coherent diffraction imaging [ 61 ], and video compressive sensing [ 62 , 63 , 64 , 65 , 66 ].…”

Deep Equilibrium Models for Snapshot Compressive Imaging

“…Our preliminary experiments found that inexact gradients could accelerate the backward passes in training our models by roughly 1.3 ∼ 1.5×. Another direction is to integrate DEQ with semantic analysis in SCI (Zhang et al 2022).…”

Switchable Diffraction Pattern Based on Cascaded Metasurfaces

“…Cascaded metasurfaces (CM) have emerged as a typical platform to provide dynamic switchable abilities, by virtue of modulating complex diffraction modes and angular spectrum through pixellevel alignment with high precision. [22][23][24][25] Some interesting optical phenomenon can be generated, for example, multilayer optical neural networks, [26][27][28][29] encryption, and decryption of optical secret sharing [30] are urgently exploited by the combination of metasurfaces. Other applications by applying orbital angular momentum multiplexing and demultiplexing, [31] switchable holographic display [32,33] have been demonstrated.…”