Learning Guided Electron Microscopy with Active Acquisition

Abstract

Single-beam scanning electron microscopes (SEM) are widely used to acquire massive data sets for biomedical study, material analysis, and fabrication inspection. Datasets are typically acquired with uniform acquisition: applying the electron beam with the same power and duration to all image pixels, even if there is great variety in the pixels' importance for eventual use. Many SEMs are now able to move the beam to any pixel in the field of view without delay, enabling them, in principle, to invest their time budget more effectively with non-uniform imaging. In this paper, we show how to use deep learning to accelerate and optimize single-beam SEM acquisition of images. Our algorithm rapidly collects an information-lossy image (e.g. low resolution) and then applies a novel learning method to identify a small subset of pixels to be collected at higher resolution based on a trade-off between the saliency and spatial diversity. We demonstrate the efficacy of this novel technique for active acquisition by speeding up the task of collecting connectomic datasets for neurobiology by up to an order of magnitude. Code is available at https://github.com/lumi9587/learning-guided-SEM.

Whole-body simulation of realistic fruit fly locomotion with deep reinforcement learning

“…In the short term, our body model and imitation learning framework can enable the model-based investigation of the neural underpinnings of sensory-motor behaviors such as escape invoked by looming stimuli [Card, 2012], gaze-stabilization [Cruz and Chiappe, 2023], the control of movement by the ventral nerve cord [Lesser et al, 2023, Azevedo et al, 2022, Takemura et al, 2023. In the long term, our whole body model, in concert with connectome-constrained deep mechanistic neural network models [Lappalainen et al, 2023, Mi et al, 2022 of the whole nervous system could eventually be used to construct whole animal models of both the entire nervous system and body of the adult fruit fly.…”

SmartEM: machine-learning guided electron microscopy

“…To mitigate 1059 dwell time contrasts and produce a visually coherent image, we 1060 applied a conditional generative adversarial network (IMAGE-1061 HOMOGENIZER, cGANs)(Mirza and Osindero, 2014). Pre-1062 vious studies used deep learning to improve the quality of mi-1063 croscopy images(Fang et al, 2021; Wang et al, 2019; Weigert 1064 Weigert et al, 2018Mi et al, 2021), de-noise EM images(Minnen 1065(Minnen et al, 2021, and perform image reconstruction across different 1066 modalities(Li et al, 2023). IMAGEHOMOGENIZER contains 1067 two convolutional neural networks (CNN): a generator and a 1068 discriminator(Isola et al, 2016).…”