Deep learning in electron microscopy

Abstract

Deep learning is transforming most areas of science and technology, including electron microscopy. This review paper offers a practical perspective aimed at developers with limited familiarity. For context, we review popular applications of deep learning in electron microscopy. Following, we discuss hardware and software needed to get started with deep learning and interface with electron microscopes. We then review neural network components, popular architectures, and their optimization. Finally, we discuss future directions of deep learning in electron microscopy.

Developing and Evaluating Deep Neural Network-Based Denoising for Nanoparticle TEM Images with Ultra-Low Signal-to-Noise

“…Convolutional neural networks (CNNs) achieve state-of-theart denoising performance on natural images (Zhang et al, 2017;Tian et al, 2019) and are an emerging tool in various fields of scientific imaging, for example, in fluorescence light microscopy (Belthangady & Royer, 2019;Zhang et al, 2019) and in medical diagnostics (Yang et al, 2017;Jifara et al, 2019). In electron microscopy, deep CNNs are rapidly being developed for denoising in a variety of applications, including structural biology (Buchholz et al, 2019;Bepler et al, 2020), semiconductor metrology (Chaudhary et al, 2019;Giannatou et al, 2019), and drift correction (Vasudevan & Jesse, 2019), among others (Ede & Beanland, 2019;Lee et al, 2020;Wang et al, 2020;Lin et al, 2021;Spurgeon et al, 2021), as highlighted in a recent review (Ede, 2020). CNNs trained for segmentation have also been used to locate the position of atomic columns (Lin et al, 2021) as well as to estimate their occupancy (Madsen et al, 2018) in relatively high SNR (S)TEM images (i.e., SNR = ∼10).…”

Deep learning detection of nanoparticles and multiple object tracking of their dynamic evolution during in situ ETEM studies

“…Recently, machine learning based methods and particularly deep learning based ones have been proposed as an accurate and efficient alternative to traditional methods 39 , 40 . Thus, particle detection and segmentation can be addressed using so-called semantic segmentation, which refers to the task of classifying all pixels of an image in a predefined semantic class (typically object or non-object for binary classification) 41 . Deep learning based methods allow to determine, using only the training data and its known ground truth, all the parameters of the network which minimizes a loss related to the task at hand 41 .…”

“…Thus, particle detection and segmentation can be addressed using so-called semantic segmentation, which refers to the task of classifying all pixels of an image in a predefined semantic class (typically object or non-object for binary classification) 41 . Deep learning based methods allow to determine, using only the training data and its known ground truth, all the parameters of the network which minimizes a loss related to the task at hand 41 . Quite naturally, the popular semantic segmentation architecture U-Net 42 has been proposed in various contexts 43 as liquid-phase TEM 44 or high-resolution TEM 45 .…”

In Situ Scanning Transmission Electron Microscopy Study of MoS2 Formation on Graphene with a Deep-Learning Framework

“…Deep learning has been solving challenges in electron microscopy. 26 Recently, deep learning of the simulation data set recognized the local atomic structure in high-resolution TEM (HRTEM) images. 27 Another deep learning using experimental HRTEM data set has been developed to reconstruct the exit wave from a single image with real microscope’s aberration parameters from the so-called Zemlin tableau method.…”