Analyzing complex single-molecule emission patterns with deep learning

Zhang, Peiyi; Liu, Sheng; Chaurasia, Abhishek; Ma, Dongsheng; Mlodzianoski, Michael J.; Culurciello, Eugenio; Huang, Fang

doi:10.1038/s41592-018-0153-5

Cited by 87 publications

(84 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DNNs are versatile tools for various applications, among which image analysis for general purpose feature recognition as well as for optical microscopy are prominent. (22)(23)(24)(25)(26) Recently, the U-net architecture has been demonstrated to be well suited for image segmentation. (27,28) Fundamentally, image segmentation is similar to sBG estimation: A featurethe PSF without BGis overlaid with the sBG, which should be identified from the combined image in order to subsequently remove it.…”

Section: Resultsmentioning

confidence: 99%

Accurate and rapid background estimation in single-molecule localization microscopy using the deep neural network BGnet

Möckl

Roy

Petrov

et al. 2019

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Background fluorescence, especially when it exhibits undesired spatial features, is a primary factor for reduced image quality in optical microscopy. Structured background is particularly detrimental when analyzing single-molecule images for 3D localization microscopy or single-molecule tracking. Here, we introduce BGnet, a deep neural network with a U-net-type architecture, as a general method to rapidly estimate the background underlying the image of a point source with excellent accuracy, even when point spread function (PSF) engineering is in use to create complex PSF shapes. We trained BGnet to extract the background from images of various PSFs and show that the identification is accurate for a wide range of different interfering background structures constructed from many spatial frequencies. Furthermore, we demonstrate that the obtained background-corrected PSF images, both for simulated and experimental data, lead to a substantial improvement in localization precision. Finally, we verify that structured background estimation with BGnet results in higher quality of super-resolution reconstructions of biological structures. Significance:A main factor that degrades image quality in fluorescence microscopy is unwanted background fluorescence. Background is almost never uniform, especially in complex samples. Rather, background usually exhibits some structure, making it very difficult to distinguish from the signal of interest. Due to this challenging problem, background fluorescence is often assumed to be uniform even though it is not. This assumption leads to deteriorated image quality, e.g. in localization-based superresolution microscopy, and to the introduction of uncharacterized biases. To overcome this challenge, we developed a general framework rooted in deep learning to accurately and rapidly estimate arbitrarily structured background using several distinct image shapes for the single emitter. Proper background estimation allows for critical performance improvement of various optical microscopy methods. Main Text:

show abstract

Section: Resultsmentioning

confidence: 99%

Accurate and rapid background estimation in single-molecule localization microscopy using the deep neural network BGnet

Möckl

Roy

Petrov

et al. 2019

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

show abstract

“…Recent advances in machine learning (ML) and specifically deep learning (DL) (27), have radically improved our capacity to access and extract information from abstract and noisy inputs independently of human interventions as we (28) and others have shown (29)(30)(31)(32)(33)(34)(35)(36). DL implementations are providing high level robust performances and have been successfully used to analyze and augment a wide range of the fluorescence microscopy analysis pipeline including assessing microscope image quality (37), in-silico cell labeling (31), single cell morphology analysis (32,34), detecting single molecules (38) and linking smFRET experiments with molecular dynamics simulations (39), amongst others (29)(30)(31)(32)(33)(34)(35)(36).…”

Section: Introductionmentioning

confidence: 99%

“…Deep learning-based analysis has several advantages over other approaches: It recognizes abstract patterns and learn useful features directly from the raw input data which allows implementation of analysis routines that don't require extensive data preprocessing or empirically defined rules and thus offer reproducible and less opinionated evaluation of single molecule data; It is significant faster than human annotation for large single molecule data sets; it comes close to, or outperforms human performance; and its performance is increased when increasing data set size constituting an ideal case for evaluating the large data sets obtained from single molecule data (29)(30)(31)(32)(33)(34)(35)(36). Especially important are convolutional DNN which learn how to best recognize particular aspects of the given data through several rounds of optimization.…”

Section: Introductionmentioning

confidence: 99%

DeepFRET: Rapid and automated single molecule FRET data classification using deep learning

Thomsen

Sletfjerding

Stella

et al. 2020

Preprint

View full text Add to dashboard Cite

Single molecule Förster Resonance energy transfer (smFRET) is a mature and adaptable method for studying the structure of biomolecules and integrating their dynamics into structural biology. The development of high throughput methodologies and the growth of commercial instrumentation have outpaced the development of rapid, standardized, and fully automated methodologies to objectively analyze the wealth of produced data. Here we present DeepFRET, an automated standalone solution based on deep learning, where the only crucial human intervention in transiting from raw microscope images to histogram of biomolecule behavior, is a user-adjustable quality threshold. Integrating all standard features of smFRET analysis, DeepFRET will consequently output common kinetic information metrics for biomolecules. We validated the utility of DeepFRET by performing quantitative analysis on simulated, ground truth, data and real smFRET data. The accuracy of classification by DeepFRET outperformed human operators and current commonly used hard threshold and reached >95% precision accuracy only requiring a fraction of the time (<1% as compared to human operators) on ground truth data. Its flawless and rapid operation on real data demonstrates its wide applicability. This level of classification was achieved without any preprocessing or parameter setting by human operators, demonstrating DeepFRET’s capacity to objectively quantify biomolecular dynamics. The provided a standalone executable based on open source code capitalises on the widespread adaptation of machine learning and may contribute to the effort of benchmarking smFRET for structural biology insights.

show abstract

“…In recent years, deep learning has been employed with great success in a variety of tasks [21], including designing optical systems [22][23][24] and interpreting single-molecule data to produce super-resolution images [25][26][27]. The multi-layer architecture of neural nets allows for extraction of complex features from data, while distilling the desired information from an input.…”

Section: Introductionmentioning

confidence: 99%

Multicolor localization microscopy and point-spread-function engineering by deep learning

Park

Shoman

et al. 2019

Opt. Express

View full text Add to dashboard Cite

Deep learning has become an extremely effective tool for image classification and image restoration problems. Here, we apply deep learning to microscopy and demonstrate how neural networks can exploit the chromatic dependence of the point-spread function to classify the colors of single emitters imaged on a grayscale camera. While existing localization microscopy methods for spectral classification require additional optical elements in the emission path, e.g., spectral filters, prisms, or phase masks, our neural net correctly identifies static and mobile emitters with high efficiency using a standard, unmodified singlechannel configuration. Furthermore, we show how deep learning can be used to design new phase-modulating elements that, when implemented into the imaging path, result in further improved color differentiation between species, including simultaneously differentiating four species in a single image.

show abstract

Analyzing complex single-molecule emission patterns with deep learning

Cited by 87 publications

References 27 publications

Accurate and rapid background estimation in single-molecule localization microscopy using the deep neural network BGnet

Accurate and rapid background estimation in single-molecule localization microscopy using the deep neural network BGnet

DeepFRET: Rapid and automated single molecule FRET data classification using deep learning

Multicolor localization microscopy and point-spread-function engineering by deep learning

Contact Info

Product

Resources

About