Content-Aware Image Restoration: Pushing the Limits of Fluorescence Microscopy

Abstract

Fluorescence microscopy is a key driver of discoveries in the life-sciences, with observable phenomena being limited by the optics of the microscope, the chemistry of the fluorophores, and the maximum photon exposure tolerated by the sample. These limits necessitate tradeoffs between imaging speed, spatial resolution, light exposure, and imaging depth. In this work we show how deep learning enables biological observations beyond the physical limitations of microscopes. On seven concrete examples we illustrate how microscopy images can be restored even if 60-fold fewer photons are used during acquisition, how isotropic resolution can be achieved even with a 10-fold under-sampling along the axial direction, and how diffraction-limited structures can be resolved at 20-times higher frame-rates compared to state-of-the-art methods. All developed image restoration methods are freely available as open source software.

Signal, noise and resolution in linear and nonlinear structured‐illumination microscopy

“…maximum likelihood based) approaches constrained to positive density and considering the Poisson statistics of photon noise and the spatially varying variance may achieve better reconstruction results with enhanced signal to noise ratios. Especially dedicated regularisation, for example, based on the spatial Hessian matrix (Huang et al, 2018) or trained artificial neural networks (Weigert et al, 2018), can achieve significant SNR advantages, yet one may expect similar SNR scaling laws to apply.…”

An overview of state‐of‐the‐art image restoration in electron microscopy

“…For higher noise levels however, deconvolution becomes much more difficult and most methods yield similar results. Weigert et al, 2017). y = Hx + n and consequently h(x; y) = Hx − y 2 2 .…”

“…These images are substantially different as they contain much more detailed and textured structures, which are harder to restore. A better approach for these methods is to train the CNN on actual EM data as in Weigert et al (2017). Typically, training a CNN from scratch requires large amounts of data and compute power.…”

“…In order to properly apply dictionary-based techniques or CNNs for EM image restoration, we recommend to extract the features from EM data (e.g. as in Weigert et al, 2017). etc. A typical denoising architecture consists of a contractive and expansive path.…”

Cyt‐Geist: Current and Future Challenges in Cytometry: Reports of the CYTO 2018 Conference Workshops

“…Recent publications using deep learning for reconstruction of fluorescence microscopy images (135,136) hold huge promise for the future. Such algorithms can reconstruct high quality images from noisy raw images acquired at low illumination levels.…”