Noise2Atom: Unsupervised Denoising for Scanning Transmission Electron Microscopy Images

Abstract

We propose an effective deep learning model to denoise scanning transmission electron microscopy (STEM) image series, named Noise2Atom, to map images from a source domain S to a target domain C, where S is for our noisy experimental dataset, and C is for the desired clear atomic images. Noise2Atom uses two external networks to apply additional constraints from the domain knowledge. This model requires no signal prior, no noise model estimation, and no paired training images. The only assumption is that the inputs are acquired with identical experimental configurations. To evaluate the restoration performance of our model, as it is impossible to obtain ground truth for our experimental dataset, we propose consecutive structural similarity (CSS) for image quality assessment, based on the fact that the structures remain much the same as the previous frame(s) within small scan intervals. We demonstrate the superiority of our model by providing evaluation in terms of CSS and visual quality on different experimental datasets.

Developing and Evaluating Deep Neural Network-Based Denoising for Nanoparticle TEM Images with Ultra-Low Signal-to-Noise

“…Convolutional neural networks (CNNs) achieve state-of-theart denoising performance on natural images (Zhang et al, 2017;Tian et al, 2019) and are an emerging tool in various fields of scientific imaging, for example, in fluorescence light microscopy (Belthangady & Royer, 2019;Zhang et al, 2019) and in medical diagnostics (Yang et al, 2017;Jifara et al, 2019). In electron microscopy, deep CNNs are rapidly being developed for denoising in a variety of applications, including structural biology (Buchholz et al, 2019;Bepler et al, 2020), semiconductor metrology (Chaudhary et al, 2019;Giannatou et al, 2019), and drift correction (Vasudevan & Jesse, 2019), among others (Ede & Beanland, 2019;Lee et al, 2020;Wang et al, 2020;Lin et al, 2021;Spurgeon et al, 2021), as highlighted in a recent review (Ede, 2020). CNNs trained for segmentation have also been used to locate the position of atomic columns (Lin et al, 2021) as well as to estimate their occupancy (Madsen et al, 2018) in relatively high SNR (S)TEM images (i.e., SNR = ∼10).…”

Review: Deep Learning in Electron Microscopy

“…For example, applications of a DNN trained with artificially deteriorated TEM images are shown in figure 1. However, ANNs have also been trained with unpaired datasets of lowquality and high-quality electron micrographs 182 , or pairs of low-quality electron micrographs 183,184 . Another approach is Noise2Void 168 , ANNs are trained from single noisy images.…”

“…However, Noise2Void removes information by masking noisy input pixels corresponding to target output pixels. So far, most ANNs that improve electron microscope signal-to-noise have been trained to decrease statistical noise 70,177,179,180,[180][181][182][183]185 . Nevertheless, ANNs have been developed for aberration correction of optical microscopy [186][187][188][189][190][191] and photoacoustic 192 signals, and to correct electron microscope scan distortions 193,194 and specimen drift 141,194,195 .…”