Leveraging generative adversarial networks to create realistic scanning transmission electron microscopy images

Abstract

The rise of automation and machine learning (ML) in electron microscopy has the potential to revolutionize materials research through autonomous data collection and processing. A significant challenge lies in developing ML models that rapidly generalize to large data sets under varying experimental conditions. We address this by employing a cycle generative adversarial network (CycleGAN) with a reciprocal space discriminator, which augments simulated data with realistic spatial frequency information. This allows the CycleGAN to generate images nearly indistinguishable from real data and provide labels for ML applications. We showcase our approach by training a fully convolutional network (FCN) to identify single atom defects in a 4.5 million atom data set, collected using automated acquisition in an aberration-corrected scanning transmission electron microscope (STEM). Our method produces adaptable FCNs that can adjust to dynamically changing experimental variables with minimal intervention, marking a crucial step towards fully autonomous harnessing of microscopy big data.

FPGA and computer-vision-based atom tracking technology for scanning probe microscopy

“…Hardware implementation can also be applied to other SPM systems. [10,31] Furthermore, the software can be implemented in other microscopy fields, such as optical microscopy, electron microscopy, [32][33][34] and so on.…”

Development of a Method for Commercial Style Transfer of Historical Architectural Facades Based on Stable Diffusion Models

“…Sun et al [9] and Ali & Lee [14] have developed tools based on GANs for urban renewal, preserving unique facade designs in urban districts. Khan et al [15] employed GANs to generate realistic Urban Mobility Networks (MoGAN), thus depicting the entirety of mobility flows within a city. Hence, GANs can, to a certain extent, enhance the efficiency of facade design during the initial phases of design projects.…”

Automated Generation and Analysis of Molecular Images Using Generative Artificial Intelligence Models

“…We note a recent work applying CycleGAN-based model for creating realistic scanning transmission electron microscopy images. 35 Both our manuscript and the referenced work explore the application of CycleGAN in the domain of highresolution imaging; our study is distinct in its focus on STM molecular images and its specialized architecture and optimization. It is also noteworthy that the regeneration of molecular images/models through our CycleGAN model achieves a throughput of 3 images per second with a standard consumer-grade graphics processing unit (GPU), beyond the speed of STM imaging under typical experimental conditions.…”

“…In contrast to traditional ML techniques, generative models output new content that shares characteristics with the training data. The generative networks, such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), are employed for image synthesis, image editing, and image-to-image translation. − Notably, GAN has been successfully used to design new molecules with specified properties. − The development of CycleGAN, a variant of GAN, involves automatic training between two different domains without the need for paired training data. CycleGAN does not compromise to have a complicated structure while still achieving striking results on a range of applications of materials science such as segmenting and postprocessing image data from scanning transmission microscopes and simulating inelastic neutron scattering data .…”

“…We note a recent work applying CycleGAN-based model for creating realistic scanning transmission electron microscopy images . Both our manuscript and the referenced work explore the application of CycleGAN in the domain of high-resolution imaging; our study is distinct in its focus on STM molecular images and its specialized architecture and optimization.…”