Noise2Atom: Unsupervised Denoising for Scanning Transmission Electron Microscopy Images

Wang, Feng; Henninen, Trond R.; Keller, Debora; Erni, Rolf

doi:10.21203/rs.3.rs-54657/v2

Cited by 12 publications

(12 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Convolutional neural networks (CNNs) achieve state-of-theart denoising performance on natural images (Zhang et al, 2017;Tian et al, 2019) and are an emerging tool in various fields of scientific imaging, for example, in fluorescence light microscopy (Belthangady & Royer, 2019;Zhang et al, 2019) and in medical diagnostics (Yang et al, 2017;Jifara et al, 2019). In electron microscopy, deep CNNs are rapidly being developed for denoising in a variety of applications, including structural biology (Buchholz et al, 2019;Bepler et al, 2020), semiconductor metrology (Chaudhary et al, 2019;Giannatou et al, 2019), and drift correction (Vasudevan & Jesse, 2019), among others (Ede & Beanland, 2019;Lee et al, 2020;Wang et al, 2020;Lin et al, 2021;Spurgeon et al, 2021), as highlighted in a recent review (Ede, 2020). CNNs trained for segmentation have also been used to locate the position of atomic columns (Lin et al, 2021) as well as to estimate their occupancy (Madsen et al, 2018) in relatively high SNR (S)TEM images (i.e., SNR = ∼10).…”

Section: Introductionmentioning

confidence: 99%

Developing and Evaluating Deep Neural Network-Based Denoising for Nanoparticle TEM Images with Ultra-Low Signal-to-Noise

Vincent

Manzorro

Mohan

et al. 2021

Microscopy and Microanalysis

View full text Add to dashboard Cite

A deep convolutional neural network has been developed to denoise atomic-resolution transmission electron microscope image datasets of nanoparticles acquired using direct electron counting detectors, for applications where the image signal is severely limited by shot noise. The network was applied to a model system of CeO2-supported Pt nanoparticles. We leverage multislice image simulations to generate a large and flexible dataset for training the network. The proposed network outperforms state-of-the-art denoising methods on both simulated and experimental test data. Factors contributing to the performance are identified, including (a) the geometry of the images used during training and (b) the size of the network's receptive field. Through a gradient-based analysis, we investigate the mechanisms learned by the network to denoise experimental images. This shows that the network exploits both extended and local information in the noisy measurements, for example, by adapting its filtering approach when it encounters atomic-level defects at the nanoparticle surface. Extensive analysis has been done to characterize the network's ability to correctly predict the exact atomic structure at the nanoparticle surface. Finally, we develop an approach based on the log-likelihood ratio test that provides a quantitative measure of the agreement between the noisy observation and the atomic-level structure in the network-denoised image.

show abstract

Section: Introductionmentioning

confidence: 99%

Developing and Evaluating Deep Neural Network-Based Denoising for Nanoparticle TEM Images with Ultra-Low Signal-to-Noise

Vincent

Manzorro

Mohan

et al. 2021

Microscopy and Microanalysis

View full text Add to dashboard Cite

show abstract

“…These methods are called full-reference, emphasizing the need for high-quality reference data [28, 45]. SSIM and consecutive similarity (CSS) metric, which is a variation of SSIM [46], are in this category. We used (local) SSIM, which provides an error map by structurally comparing the reconstructed image with the reference image, and based on that error map controlled for reconstruction defects.…”

Section: Discussionmentioning

confidence: 99%

“…The first, second and the third term are respectively called luminance (mean), contrast (standard deviation) and structural (covariance). As the standard deviation of an image is not usually affected by denoising [46], in our experiment, we only focused on luminance and structural term and we investigated how luminance and structural terms of SSIM depend on the duration of photon collection in integrating versus averaging mode detectors. We observed that, for integrating detectors, the mean term influences the SSIM value, so that structural reliability cannot be directly compared if this term is included in the calculation of the SSIM value (Fig.…”

Section: Methodsmentioning

confidence: 99%

Deep-learning microscopy image reconstruction with quality control reveals second-scale rearrangements in RNA polymerase II clusters

Hajiabadi

Mamontova

Prizak

et al. 2021

Preprint

View full text Add to dashboard Cite

Fluorescence microscopy, a central tool of biological research, is subject to inherent trade-offs in experiment design. For instance, image acquisition speed can only be increased in exchange for a lowered signal quality, or for an increased rate of photo-damage to the specimen. Computational denoising can recover some loss of signal, extending the trade-off margin for high-speed imaging. Recently proposed denoising on the basis of neural networks shows exceptional performance but raises concerns of errors typical of neural networks. Here, we present a work-flow that supports an empirically optimized reduction of exposure times, as well as per-image quality control to exclude images with reconstruction errors. We implement this work-flow on the basis of the denoising tool Noise2Void and assess the molecular state and three-dimensional shape of RNA Polymerase II (Pol II) clusters in live zebrafish embryos. Image acquisition speed could be tripled, achieving 2-second time resolution and 350-nanometer lateral image resolution. The obtained data reveal stereotyped events of approximately 10 seconds duration: initially, the molecular mark for initiated Pol II increases, then the mark for active Pol II increases, and finally Pol II clusters take on a stretched and unfolded shape. An independent analysis based on fixed sample images reproduces this sequence of events, and suggests that they are related to the transient association of genes with Pol II clusters. Our work-flow consists of procedures that can be implemented on commercial fluorescence microscopes without any hardware or software modification, and should therefore be transferable to many other applications.

show abstract

“…In the case of the TEM data in our case study, unsupervised denoising does not perform well (see Section 6, and in particular Figures 16, and 17), possibly due to the SNR, which is orders of magnitude lower (around 3 dB) than that reported in these works (around 27 dB for Zhang et al (2019)). Improving the performance of unsupervised denoising methods at low SNRs is thus an important topic for future research - Wang et al (2020) have reported promising results in this direction for scanning TEM.…”

Section: Related Workmentioning

confidence: 99%

Deep Denoising For Scientific Discovery: A Case Study In Electron Microscopy

Mohan¹,

Manzorro²,

Vincent³

et al. 2020

Preprint

View full text Add to dashboard Cite

Denoising is a fundamental challenge in scientific imaging. Deep convolutional neural networks (CNNs) provide the current state of the art in denoising natural images, where they produce impressive results. However, their potential has barely been explored in the context of scientific imaging. Denoising CNNs are typically trained on real natural images artificially corrupted with simulated noise. In contrast, in scientific applications, noiseless ground-truth images are usually not available. To address this issue, we propose a simulation-based denoising (SBD) framework, in which CNNs are trained on simulated images. We test the framework on data obtained from transmission electron microscopy (TEM), an imaging technique with widespread applications in material science, biology, and medicine. SBD outperforms existing techniques by a wide margin on a simulated benchmark dataset, as well as on real data. Apart from the denoised images, SBD generates likelihood maps to visualize the agreement between the structure of the denoised image and the observed data. Our results reveal shortcomings of stateof-the-art denoising architectures, such as their small field-of-view: substantially increasing the field-of-view of the CNNs allows them to exploit non-local periodic patterns in the data, which is crucial at high noise levels. In addition, we analyze the generalization capability of SBD, demonstrating that the trained networks are robust to variations of imaging parameters and of the underlying signal structure. Finally, we release the first publicly available benchmark dataset of TEM images, containing 18,000 examples.

show abstract

Noise2Atom: Unsupervised Denoising for Scanning Transmission Electron Microscopy Images

Cited by 12 publications

References 20 publications

Developing and Evaluating Deep Neural Network-Based Denoising for Nanoparticle TEM Images with Ultra-Low Signal-to-Noise

Developing and Evaluating Deep Neural Network-Based Denoising for Nanoparticle TEM Images with Ultra-Low Signal-to-Noise

Deep-learning microscopy image reconstruction with quality control reveals second-scale rearrangements in RNA polymerase II clusters

Deep Denoising For Scientific Discovery: A Case Study In Electron Microscopy

Contact Info

Product

Resources

About