A “Human-in-the-Loop” Approach for Semi-automated Image Restoration in Electron Microscopy

Abstract

The recent advent of 3D in Electron Microscopy (EM) has allowed for detection of detailed sub-cellular nanometer resolution structures. While being a scientific breakthrough, this has also caused an explosion in dataset size, necessitating the development of automated workflows. Automated workflows typically benefit reproducibility and throughput compared to manual analysis. The risk of automation is that it ignores the expertise of the microscopy user that comes with manual analysis. To mitigate this risk, this paper presents a hybrid paradigm. We propose a 'human-in-the-loop' (HITL) model that combines expert microscopy knowledge with the power of large-scale parallel computing to improve EM image quality through advanced image restoration algorithms. An interactive graphical user interface, publicly available as an ImageJ plugin, was developed to allow biologists to use our framework in an intuitive and user-friendly fashion. We show that this plugin improves visualization of EM ultrastructure and subsequent (semi-)automated segmentation and image analysis. segmentation is done manually 12 , which has several issues. Firstly, such a process is very labor-intensive, making it slow and costly. Secondly, reproducibility is an issue since humans tends to be subjective and biased, and different annotators are likely to produce different segmentations. Lastly, segmentation quality depends on high-quality data, prompting the introduction of quality control.Automated analysis is key to solving the bottleneck in the entire 3D EM workflow. Current advances in large-scale computing and computer vision [13][14][15] can significantly speed up the whole automated image analysis process and make it more cost efficient, potentially leading to challenging and successful research projects that were never possible before 16 . Furthermore, automated methods typically have a formal mathematical formulation and thus the potential to be less biased and more reproducible. Recent automated analysis frameworks are all focused on the segmentation process as such, by either facilitating the manual tracing 17,18 or providing algorithms that allow for interactive learning 19 . As data quality control is omitted, these methods often underperform when challenged with image artifacts such as blur and noise. In practice, this is fairly common in volume EM considering the physical and theoretical issues, including imperfect lenses, thermal heating, diffraction and electron counting errors 20 . Moreover, these artifacts tend to be more pronounced for high throughput data acquisition, which is typically desired in challenging projects 16 . Recent advances in image denoising and deconvolution can significantly mitigate these artifacts 21 .The field of image restoration is a popular field in general image processing: state-of-the-art methods are based on multiresolution shrinkage 22, 23 , non-local pixel averaging 23, 24 , Bayesian estimation 25,26 or convolutional neural networks 27 . Even though most of these methods are available to the communit...

A Survey of Human-in-the-loop for Machine Learning

“…• It is necessary to pay attention to predictive parameter optimization based on supervised regression models and scientifically analyzing correlations between the parameters of different algorithms for HITL image restoration [127].…”

“…For instance, in the field of Electron Microscopy, one drawback of automation is that it generally ignores the expertise of the microscopy user that comes with manual analysis. To alleviate such a challenging problem, Roels et al [127] propose a hybrid HITL system that incorporates expert microscopy knowledge with the power of large-scale parallel computing to enhance the Electron Microscopy image quality by exploiting image restoration algorithms. The HITL workflow in the system consists of six steps and the training and inference interaction between individuals and the framework in the region of interest selection and interactive parameter optimization.…”

“…Associating the contents in Table 2, this phenomenon in CV is similar to NLP, which demonstrates that the core motivation of almost all HITL studies in both CV and NLP serves the purpose of boosting model performance. We also notice that in Table 3, only one work [127] tries to bring interpretability for the model. Roels et al [127] have validated the potential enhancements that DenoisEM can provide in 3D EM image interpretation by denoising SBF-SEM image data of an Arabidopsis thaliana root tip.…”

“…We also notice that in Table 3, only one work [127] tries to bring interpretability for the model. Roels et al [127] have validated the potential enhancements that DenoisEM can provide in 3D EM image interpretation by denoising SBF-SEM image data of an Arabidopsis thaliana root tip. Besides, HITL conception can also improve the usability of CV models.…”

Tonic GABAAReceptor-Mediated Currents of Human Cortical GABAergic Interneurons Vary Amongst Cell Types

“…Spectral unmixing was then carried out using these spectra to separate the streptavidin and lipofuscin emissions into two pseudochannels. Denoising of the resulting image was carried out using the non-local means option in DenoiseEM (Roels et al, 2019).…”