Implementing an accurate and rapid sparse sampling approach for low-dose atomic resolution STEM imaging

Compressive sensing approaches are beginning to take hold in (scanning) transmission electron microscopy (S/TEM) [1,2,3]. Compressive sensing is a mathematical theory about acquiring signals in a compressed form (measurements) and the probability of recovering the original signal by solving an inverse problem [4]. The inverse problem is underdetermined (more unknowns than measurements), so it is not obvious that recovery is possible. Compression is achieved by taking inner products of the signal with measurement weight vectors. Both Gaussian random weights and Bernoulli (0,1) random weights form a large class of measurement vectors for which recovery is possible. The measurements can also be designed through an optimization process. The key insight for electron microscopists is that compressive sensing can be used to increase acquisition speed and reduce dose.Building on work initially developed for optical cameras, this new paradigm will allow electron microscopists to solve more problems in the engineering and life sciences. We will be collecting orders of magnitude more data than previously possible. The reason that we will have more data is because we will have increased temporal/spatial/spectral sampling rates, and we will be able to interrogate larger classes of samples that were previously too beam sensitive to survive the experiment. For example consider an in-situ experiment that takes 1 minute. With traditional sensing, we might collect 5 images per second for a total of 300 images. With compressive sensing, each of those 300 images can be expanded into 10 more images, making the collection rate 50 images per second, and the decompressed data a total of 3000 images [3].But, what are the implications, in terms of data, for this new methodology? Acquisition of compressed data will require downstream reconstruction to be useful. The reconstructed data will be much larger than traditional data, we will need space to store the reconstructions during analysis, and the computational demands for analysis will be higher. Moreover, there will be time costs associated with reconstruction.Deep learning [5] is an approach to address these problems. Deep learning is a hierarchical approach to find useful (for a particular task) representations of data. Each layer of the hierarchy is intended to represent higher levels of abstraction. For example, a deep model of faces might have sinusoids, edges and gradients in the first layer; eyes, noses, and mouths in the second layer, and faces in the third layer. There has been significant effort recently in deep learning algorithms for tasks beyond image classification such as compressive reconstruction [6] and image segmentation [7]. A drawback of deep learning, however, is that training the model requires large datasets and dedicated computational resources (to reduce training time to a few days). A second issue is that deep learning is not userfriendly and the meaning behind the results is usually not interpretable. We have shown it is possible to reduce the data set size ...

“…

…”

mentioning

confidence: 99%

Less is More: Bigger Data from Compressive Measurements

Stevens

Browning

2017

“…decreased sensitivity to thickness effects) makes it the ideal method to image through the ~100-500 nm thick in-situ liquid cells that are typically used. Somewhat counter intuitively, the STEM imaging process is also optimized for controlling and reducing beam damage -the dose is controlled by the beam size and dwell time of the scan, while the probe only illuminates a small area, thereby reducing heating and depletion effects [1].The use of the scanned beam to form images has another advantage in that it can readily make use of compressive sensing/in-painting approaches [2,3] to reduce the overall dose during the experiment and increase the acquisition speed [4,5]. The in-painting method is a mechanism by which a small sub-set of pixels can be acquired experimentally and then mathematical processes used to in-paint the missing information.…”

mentioning

confidence: 99%

“…The use of the scanned beam to form images has another advantage in that it can readily make use of compressive sensing/in-painting approaches [2,3] to reduce the overall dose during the experiment and increase the acquisition speed [4,5]. The in-painting method is a mechanism by which a small sub-set of pixels can be acquired experimentally and then mathematical processes used to in-paint the missing information.…”

mentioning

confidence: 99%

“…The in-painting method is a mechanism by which a small sub-set of pixels can be acquired experimentally and then mathematical processes used to in-paint the missing information. Demonstration of this approach in practical acquisition of images [5] indicates that full reconstruction can be obtained for sub-sampling down to the ~10% level. In addition to reconstructing the images, analytics to quantify the content of the images has also been shown to work for extremely low sampling rates [6].…”

mentioning

confidence: 99%

See 1 more Smart Citation

Controlling the Reaction Process in Operando STEM by Pixel Sub-Sampling

Mehdi¹,

Stevens²,

Kovařík³

et al. 2017

Recently there has been an increase in the number of experiments making use of either in-situ gas or liquid stages, or using dedicated environmental (scanning) transmission electron microscopes (S/TEM) to study dynamic materials processes. While in-situ observations have traditionally been performed in TEM mode, allowing the intrinsic increase in temporal resolution of the projection image to be utilized, there are a number of key advantages of using the STEM imaging mode for these experiments. Namely, the same physics that makes high angle annular dark field (HAADF) or Z-contrast imaging optimum for quantifying small metal/oxide catalyst particles on a support also makes it optimum for imaging particle dynamics in liquids (now the liquid is the background in the image rather than the support). In addition, the incoherence of the Z-contrast image (i.e. decreased sensitivity to thickness effects) makes it the ideal method to image through the ~100-500 nm thick in-situ liquid cells that are typically used. Somewhat counter intuitively, the STEM imaging process is also optimized for controlling and reducing beam damage -the dose is controlled by the beam size and dwell time of the scan, while the probe only illuminates a small area, thereby reducing heating and depletion effects [1].The use of the scanned beam to form images has another advantage in that it can readily make use of compressive sensing/in-painting approaches [2,3] to reduce the overall dose during the experiment and increase the acquisition speed [4,5]. The in-painting method is a mechanism by which a small sub-set of pixels can be acquired experimentally and then mathematical processes used to in-paint the missing information. Demonstration of this approach in practical acquisition of images [5] indicates that full reconstruction can be obtained for sub-sampling down to the ~10% level. In addition to reconstructing the images, analytics to quantify the content of the images has also been shown to work for extremely low sampling rates [6]. Such results suggest that an extra degree of flexibility in performing in-situ liquid or gas experiments can be built in by changing the level of sampling, i.e changing the density of dosed pixels used to acquire the images. Figure 1 shows the final fully sampled images obtained after in-situ liquid nucleation and growth experiments were performed on a 10 mM AgNO 3 solution sampled in full mode ( Fig 1A) and 12% subsampled mode (Fig. 1B). What can be clearly seen from these results that the sub-pixel sampling leads to different growth morphology of silver nanoparticles. In this presentation, a discussion on how subsampling affects the growth mode for this and other in-situ liquid experiments will be discussed in detail [7].

“…sampling a full diffraction image at each location in a 2-D grid across the sample). This approach has been employed experimentally for materials science samples requiring low-dose imaging [2], and can be readily applied to biological samples. Figure 1 shows the resolution possible in a complex biological system, mouse pancreatic islet beta cells [4], when tomogram slices are reconstructed using subsampling.…”

mentioning

confidence: 99%

Cryo-STEM Tomography with Inpainting

Stevens¹,

Browning²

2017

Traditionally, microscopists have worked with the Nyquist-Shannon theory of sampling, which states that to be able to reconstruct the image fully it needs to be sampled at a rate of at least twice the highest spatial frequency in the image. This sampling rate assumes that the image is sampled at regular intervals and that every pixel contains information that is crucial for the image (it even assumes that noise is important). Images in general, and especially low dose S/TEM images, contain significantly less information than can be encoded by a grid of pixels (which is why image compression works). Mathematically speaking, the image data has a low dimensional or sparse representation. Through the application of compressive sensing methods [1,2,3] this representation can be found using pre-designed measurements that are usually random for implementation simplicity. These measurements and the compressive sensing reconstruction algorithms have the added benefit of reducing noise.