Super-compression of large electron microscopy time series by deep compressive sensing learning

Zheng, Siming; Wang, Chunyang; Yuan, Xin; Xin, Huolin L.

doi:10.1016/j.patter.2021.100292

Cited by 22 publications

(8 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition to spectral SCI reconstruction as shown in this work, we do believe our network can be used in medical images [ 59 ], image compression [ 60 ], temporal compressive coherent diffraction imaging [ 61 ], and video compressive sensing [ 62 , 63 , 64 , 65 , 66 ].…”

Section: Discussionmentioning

confidence: 99%

Hybrid Multi-Dimensional Attention U-Net for Hyperspectral Snapshot Compressive Imaging Reconstruction

Zheng

Zhu

Chen

2023

Entropy

Self Cite

View full text Add to dashboard Cite

In order to capture the spatial-spectral (x,y,λ) information of the scene, various techniques have been proposed. Different from the widely used scanning-based methods, spectral snapshot compressive imaging (SCI) utilizes the idea of compressive sensing to compressively capture the 3D spatial-spectral data-cube in a single-shot 2D measurement and thus it is efficient, enjoying the advantages of high-speed and low bandwidth. However, the reconstruction process, , to retrieve the 3D cube from the 2D measurement, is an ill-posed problem and it is challenging to reconstruct high quality images. Previous works usually use 2D convolutions and preliminary attention to address this challenge. However, these networks and attention do not exactly extract spectral features. On the other hand, 3D convolutions can extract more features in a 3D cube, but increase computational cost significantly. To balance this trade-off, in this paper, we propose a hybrid multi-dimensional attention U-Net (HMDAU-Net) to reconstruct hyperspectral images from the 2D measurement in an end-to-end manner. HMDAU-Net integrates 3D and 2D convolutions in an encoder–decoder structure to fully utilize the abundant spectral information of hyperspectral images with a trade-off between performance and computational cost. Furthermore, attention gates are employed to highlight salient features and suppress the noise carried by the skip connections. Our proposed HMDAU-Net achieves superior performance over previous state-of-the-art reconstruction algorithms.

show abstract

Section: Discussionmentioning

confidence: 99%

Hybrid Multi-Dimensional Attention U-Net for Hyperspectral Snapshot Compressive Imaging Reconstruction

Zheng

Zhu

Chen

2023

Entropy

Self Cite

View full text Add to dashboard Cite

show abstract

“…CS has also showed potential for compressing the data volume associated to time series in in situ experiments. 164 As introduced before, this shines light on the important role ML can play when dealing with in situ experimental setups, topic that will conclude this first section of the review. [165][166][167][168] In the previous paragraphs we have already reviewed some examples in which the effect of the beam, mainly on 2D materials, was carefully considered.…”

Section: Unsupervised Exploratory Routinesmentioning

confidence: 91%

Machine learning in electron microscopy for advanced nanocharacterization: current developments, available tools and future outlook

2022

View full text Add to dashboard Cite

show abstract

“…In practical applications [70] with a large field-of-view, a model for large-scale reconstruction and high inference speed is urgently needed. To this end, we employ a physics-driven two-stage model [71] for the reconstruction.…”

Section: Sci Reconstructionmentioning

confidence: 99%

From compressive sampling to compressive tasking: retrieving semantics in compressed domain with low bandwidth

et al. 2022

Self Cite

View full text Add to dashboard Cite

High-throughput imaging is highly desirable in intelligent analysis of computer vision tasks. In conventional design, throughput is limited by the separation between physical image capture and digital post processing. Computational imaging increases throughput by mixing analog and digital processing through the image capture pipeline. Yet, recent advances of computational imaging focus on the “compressive sampling”, this precludes the wide applications in practical tasks. This paper presents a systematic analysis of the next step for computational imaging built on snapshot compressive imaging (SCI) and semantic computer vision (SCV) tasks, which have independently emerged over the past decade as basic computational imaging platforms. SCI is a physical layer process that maximizes information capacity per sample while minimizing system size, power and cost. SCV is an abstraction layer process that analyzes image data as objects and features, rather than simple pixel maps. In current practice, SCI and SCV are independent and sequential. This concatenated pipeline results in the following problems: i) a large amount of resources are spent on task-irrelevant computation and transmission, ii) the sampling and design efficiency of SCI is attenuated, and iii) the final performance of SCV is limited by the reconstruction errors of SCI. Bearing these concerns in mind, this paper takes one step further aiming to bridge the gap between SCI and SCV to take full advantage of both approaches. After reviewing the current status of SCI, we propose a novel joint framework by conducting SCV on raw measurements captured by SCI to select the region of interest, and then perform reconstruction on these regions to speed up processing time. We use our recently built SCI prototype to verify the framework. Preliminary results are presented and the prospects for a joint SCI and SCV regime are discussed. By conducting computer vision tasks in the compressed domain, we envision that a new era of snapshot compressive imaging with limited end-to-end bandwidth is coming.

show abstract

Super-compression of large electron microscopy time series by deep compressive sensing learning

Cited by 22 publications

References 52 publications

Hybrid Multi-Dimensional Attention U-Net for Hyperspectral Snapshot Compressive Imaging Reconstruction

Hybrid Multi-Dimensional Attention U-Net for Hyperspectral Snapshot Compressive Imaging Reconstruction

Machine learning in electron microscopy for advanced nanocharacterization: current developments, available tools and future outlook

From compressive sampling to compressive tasking: retrieving semantics in compressed domain with low bandwidth

Contact Info

Product

Resources

About