Applying compressive sensing to TEM video: a substantial frame rate increase on any camera

Stevens, Andrew; Kovařík, Libor; Abellán, Patricia; Yuan, Xin; Carin, Lawrence; Browning, Nigel D.

doi:10.1186/s40679-015-0009-3

Cited by 64 publications

(47 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In scanning transmission electron microscopy (STEM), it is possible to compress an image spatially by reducing the number of measured pixels, which decreases electron dose and increases sensing speed [2,3,4]. The two requirements for CS to work are: (1) sparsity of basis coefficients and (2) incoherence of the sensing system and the representation system.…”

mentioning

confidence: 99%

Acquisition of STEM Images by Adaptive Compressive Sensing

et al. 2017

Self Cite

View full text Add to dashboard Cite

Compressive Sensing (CS) allows a signal to be sparsely measured first and accurately recovered later in software [1]. In scanning transmission electron microscopy (STEM), it is possible to compress an image spatially by reducing the number of measured pixels, which decreases electron dose and increases sensing speed [2,3,4]. The two requirements for CS to work are: (1) sparsity of basis coefficients and (2) incoherence of the sensing system and the representation system. However, when pixels are missing from the image, it is difficult to have an incoherent sensing matrix. Nevertheless, dictionary learning techniques such as Beta-Process Factor Analysis (BPFA) [5] are able to simultaneously discover a basis and the sparse coefficients in the case of missing pixels.On top of CS, we would like to apply active learning [6,7] to further reduce the proportion of pixels being measured, while maintaining image reconstruction quality. Suppose we initially sample 10% of random pixels. We wish to select the next 1% of pixels that are most useful in recovering the image. Now, we have 11% of pixels, and we want to decide the next 1% of "most informative" pixels. Active learning methods are online and sequential in nature. Our goal is to adaptively discover the best sensing mask during acquisition using feedback about the structures in the image. In the end, we hope to recover a high quality reconstruction with a dose reduction relative to the non-adaptive (random) sensing scheme. In doing this, we try three metrics applied to the partial reconstructions for selecting the new set of pixels: (1) variance, (2) Kullback-Leibler (K-L) divergence using a Radial Basis Function (RBF) kernel, and (3) entropy.Figs. 1 and 2 display the comparison of Peak Signal-to-Noise (PSNR) using these three different active learning methods at different percentages of sampled pixels. At 20% level, all the three active learning methods underperform the original CS without active learning. However, they all beat the original CS as more of the "most informative" pixels are sampled. One can also argue that CS equipped with active learning requires less sampled pixels to achieve the same value of PSNR than CS with pixels randomly sampled, since all the three PSNR curves with active learning grow at a faster pace than that without active learning when the pixel fraction is larger than 20%. For this particular STEM image, by observing the reconstructed images and the sensing masks, we find that while the method based on RBF kernel acquires samples more uniformly, the one on entropy samples more areas of significant change, thus less uniformly. The K-L divergence method performs the best in terms of reconstruction error (PSNR) for this example [8].

show abstract

mentioning

confidence: 99%

Acquisition of STEM Images by Adaptive Compressive Sensing

et al. 2017

Self Cite

View full text Add to dashboard Cite

show abstract

“…With traditional sensing, we might collect 5 images per second for a total of 300 images. With compressive sensing, each of those 300 images can be expanded into 10 more images, making the collection rate 50 images per second, and the decompressed data a total of 3000 images [3].…”

mentioning

confidence: 99%

“…

…”

mentioning

confidence: 99%

Less is More: Bigger Data from Compressive Measurements

Stevens

Browning

2017

Microsc Microanal

Self Cite

View full text Add to dashboard Cite

Compressive sensing approaches are beginning to take hold in (scanning) transmission electron microscopy (S/TEM) [1,2,3]. Compressive sensing is a mathematical theory about acquiring signals in a compressed form (measurements) and the probability of recovering the original signal by solving an inverse problem [4]. The inverse problem is underdetermined (more unknowns than measurements), so it is not obvious that recovery is possible. Compression is achieved by taking inner products of the signal with measurement weight vectors. Both Gaussian random weights and Bernoulli (0,1) random weights form a large class of measurement vectors for which recovery is possible. The measurements can also be designed through an optimization process. The key insight for electron microscopists is that compressive sensing can be used to increase acquisition speed and reduce dose.Building on work initially developed for optical cameras, this new paradigm will allow electron microscopists to solve more problems in the engineering and life sciences. We will be collecting orders of magnitude more data than previously possible. The reason that we will have more data is because we will have increased temporal/spatial/spectral sampling rates, and we will be able to interrogate larger classes of samples that were previously too beam sensitive to survive the experiment. For example consider an in-situ experiment that takes 1 minute. With traditional sensing, we might collect 5 images per second for a total of 300 images. With compressive sensing, each of those 300 images can be expanded into 10 more images, making the collection rate 50 images per second, and the decompressed data a total of 3000 images [3].But, what are the implications, in terms of data, for this new methodology? Acquisition of compressed data will require downstream reconstruction to be useful. The reconstructed data will be much larger than traditional data, we will need space to store the reconstructions during analysis, and the computational demands for analysis will be higher. Moreover, there will be time costs associated with reconstruction.Deep learning [5] is an approach to address these problems. Deep learning is a hierarchical approach to find useful (for a particular task) representations of data. Each layer of the hierarchy is intended to represent higher levels of abstraction. For example, a deep model of faces might have sinusoids, edges and gradients in the first layer; eyes, noses, and mouths in the second layer, and faces in the third layer. There has been significant effort recently in deep learning algorithms for tasks beyond image classification such as compressive reconstruction [6] and image segmentation [7]. A drawback of deep learning, however, is that training the model requires large datasets and dedicated computational resources (to reduce training time to a few days). A second issue is that deep learning is not userfriendly and the meaning behind the results is usually not interpretable. We have shown it is possible to reduce the data set size ...

show abstract

“…Technological limitations are also associated with obtaining images faster; in the case of the TEM low-dose imaging requirements exceed the framerate of the camera, while in STEM the hysteresis in the scan coils limits the overall scanning speed that can be obtained. To overcome some of these limitations it is possible to use a set of image reconstruction methods such as compressive sensing and in-painting [1,2] to reduce the overall number of readouts/pixels in the TEM/STEM images/movies and lower the overall electron dose while at the same time increasing the speed of the image and reducing the overall size of the dataset acquired [3,4].An example of the use of in-painting to recover pixels in a sub-sampled STEM image is shown in Figure 1. Here the image is sub-sampled after acquisition and recovered to demonstrate that high resolution images typically acquired in a TEM/STEM are over-sampled.…”

mentioning

confidence: 99%

Implementing Sub-sampling Methods for Low-Dose (Scanning) Transmission Electron Microscopy (S/TEM)

Browning

Stevens²,

Kovařík³

et al. 2017

Microsc Microanal

Self Cite

View full text Add to dashboard Cite

In many practical applications of high resolution (scanning) transmission electron microscopy, the resolution obtainable in an image is determined solely by the stability of the sample. This is particularly true in the era of aberration corrected S/TEM where the dose on the sample for the highest resolution images can easily exceed 10 5 electrons/Å 2 during routine operation. This dose sensitivity presents extreme constraints on applications such as 3-D imaging, spectroscopic imaging and in-situ experiments, where prolonged exposure to the beam can result in damage occurring before the experiment can be completed. To ensure that observations are free from damage artifacts, it is therefore essential that images are acquired under conditions where the maximum information can be extracted from the minimum amount of electron dose.Traditionally low-dose imaging is performed by either acquiring images faster or changing the emission conditions of the gun. However, when there is a change in the emission the microscope alignment changes and it is not always possible to run the automated corrector alignments in low-dose mode to obtain the highest resolution images. Technological limitations are also associated with obtaining images faster; in the case of the TEM low-dose imaging requirements exceed the framerate of the camera, while in STEM the hysteresis in the scan coils limits the overall scanning speed that can be obtained. To overcome some of these limitations it is possible to use a set of image reconstruction methods such as compressive sensing and in-painting [1,2] to reduce the overall number of readouts/pixels in the TEM/STEM images/movies and lower the overall electron dose while at the same time increasing the speed of the image and reducing the overall size of the dataset acquired [3,4].An example of the use of in-painting to recover pixels in a sub-sampled STEM image is shown in Figure 1. Here the image is sub-sampled after acquisition and recovered to demonstrate that high resolution images typically acquired in a TEM/STEM are over-sampled. Of course, such post-acquisition sub sampling does not decrease the dose on the sample. Setting up the STEM to acquire sub-sampled images directly allows an immediate benefit in terms of the number of pixels sampled (dose, speed and data transfer). Figure 2 shows an image of CaCo3 acquired using a sub-sampling STEM mode of operation [5]. Here the scan proceeds through a pixel hopping mechanism so that a small fraction of the pixels is actually acquired (this also benefits from an increase in speed of acquisition by reducing the number of scan flyback/stabilization procedures during the scan). In this presentation, the use of compressive sensing and in-painting methods to increase the flexibility and speed while decreasing the dose of all modes of TEM/STEM imaging and spectroscopy will be described. The major limitations and the potential benefits of active learning for the acquisition of dynamic movies at vastly increased framerates during in-situ measurements will als...

show abstract

Applying compressive sensing to TEM video: a substantial frame rate increase on any camera

Cited by 64 publications

References 45 publications

Acquisition of STEM Images by Adaptive Compressive Sensing

Acquisition of STEM Images by Adaptive Compressive Sensing

Less is More: Bigger Data from Compressive Measurements

Implementing Sub-sampling Methods for Low-Dose (Scanning) Transmission Electron Microscopy (S/TEM)

Contact Info

Product

Resources

About