Deep learning 2D and 3D optical sectioning microscopy using cross-modality Pix2Pix cGAN image translation

Zhuge, Huimin; Summa, Brian; Hamm, Jihun; Brown, J. Quincy

doi:10.1364/boe.439894

Cited by 9 publications

(5 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are also related works employing deep learning methods to achieve pseudo-optical sectioning images in recent years, including the use of convolutional neural networks and supervised learning to reconstruct optically sectioned images using widefield images as input and trained with spatially aligned structured illumination microscopy images ( 48 , 49 ). However, it is challenging to transform from widefield images to multiple confocal images with pseudo-optical sectioning capability when the image features vary from layer to layer using a supervised method.…”

Section: Resultsmentioning

confidence: 99%

Exceeding the limit for microscopic image translation with a deep learning-based unified framework

Dai,

Wong,

Wong

2024

PNAS Nexus

View full text Add to dashboard Cite

Deep learning algorithms have been widely used in microscopic image translation. The corresponding data-driven models can be trained by supervised or unsupervised learning depending on the availability of paired data. However, general cases are where the data are only roughly paired such that supervised learning could be invalid due to data unalignment, and unsupervised learning would be less ideal as the roughly paired information is not utilized. In this work, we propose a unified framework (U-Frame) that unifies supervised and unsupervised learning by introducing a tolerance size that can be adjusted automatically according to the degree of data misalignment. Together with the implementation of a global sampling rule, we demonstrate that U-Frame consistently outperforms both supervised and unsupervised learning in all levels of data misalignments (even for perfectly aligned image pairs) in a myriad of image translation applications, including pseudo optical sectioning, virtual histological staining (with clinical evaluations for cancer diagnosis), improvement of signal-to-noise ratio or resolution, and prediction of fluorescent labels, potentially serving as new standard for image translation.

show abstract

Section: Resultsmentioning

confidence: 99%

Exceeding the limit for microscopic image translation with a deep learning-based unified framework

Dai,

Wong,

Wong

2024

PNAS Nexus

View full text Add to dashboard Cite

show abstract

“…Generative Learning-based Cross-modality Data Translation A plethora of works performs cross-modality data translation through conditional generative adversarial networks (cGANs) [72,71,58,54,77,4]. Those cGANs-based crossmodality data translation usually requires the generation of a supervised signal for conditioning the generator network (e.g., Zhuge et.…”

Section: Related Workmentioning

confidence: 99%

Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusion

Wang¹,

Yingyu²,

Sermesant³

et al. 2023

Preprint

View full text Add to dashboard Cite

Cross-modality data translation has attracted great interest in image computing. Deep generative models (e.g., GANs) show performance improvement in tackling those problems. Nevertheless, as a fundamental challenge in image translation, the problem of Zero-shot-Learning Cross-Modality Data Translation with fidelity remains unanswered. This paper proposes a new unsupervised zero-shot-learning method named Mutual Information guided Diffusion cross-modality data translation Model (MIDiffusion), which learns to translate the unseen source data to the target domain. The MIDiffusion leverages a score-matching-based generative model, which learns the prior knowledge in the target domain. We propose a differentiable local-wise-MI-Layer (LM I) for conditioning the iterative denoising sampling. The LM I captures the identical cross-modality features in the statistical domain for the diffusion guidance; thus, our method does not require retraining when the source domain is changed, as it does not rely on any direct mapping between the source and target domains. This advantage is critical for applying cross-modality data translation methods in practice, as a reasonable amount of source domain dataset is not always available for supervised training. We empirically show the advanced performance of MIDiffusion in comparison with an influential group of generative models, including adversarial-based and other scorematching-based models.

show abstract

“…16 In other research, by focusing on virtual staining in microscopy, a cGAN was proposed to achieve reliable morphometric assessment of corneal endothelial cells, demonstrating its effectiveness with minimal post-processing requirements. 17 In another study, Nguyen et al utilised convolutional neural network models, including modified UNET with cGAN, capable of predicting focused fluorescent microscopic images and focal length, contributing to improved autofocus mechanisms in microscopy. 18 In recent years, fibre bundle endoscopy has been widely used to achieve super-resolution imaging for endoscopic applications.…”

Section: Introductionmentioning

confidence: 99%

Enhancing resolution and contrast in fibre bundle‐based fluorescence microscopy using generative adversarial network

Ketabchi,

Morova,

Uysalli

et al. 2024

Journal of Microscopy

View full text Add to dashboard Cite

Fibre bundle (FB)‐based endoscopes are indispensable in biology and medical science due to their minimally invasive nature. However, resolution and contrast for fluorescence imaging are limited due to characteristic features of the FBs, such as low numerical aperture (NA) and individual fibre core sizes. In this study, we improved the resolution and contrast of sample fluorescence images acquired using in‐house fabricated high‐NA FBs by utilising generative adversarial networks (GANs). In order to train our deep learning model, we built an FB‐based multifocal structured illumination microscope (MSIM) based on a digital micromirror device (DMD) which improves the resolution and the contrast substantially compared to basic FB‐based fluorescence microscopes. After network training, the GAN model, employing image‐to‐image translation techniques, effectively transformed wide‐field images into high‐resolution MSIM images without the need for any additional optical hardware. The results demonstrated that GAN‐generated outputs significantly enhanced both contrast and resolution compared to the original wide‐field images. These findings highlight the potential of GAN‐based models trained using MSIM data to enhance resolution and contrast in wide‐field imaging for fibre bundle‐based fluorescence microscopy.Lay Description: Fibre bundle (FB) endoscopes are essential in biology and medicine but suffer from limited resolution and contrast for fluorescence imaging. Here we improved these limitations using high‐NA FBs and generative adversarial networks (GANs). We trained a GAN model with data from an FB‐based multifocal structured illumination microscope (MSIM) to enhance resolution and contrast without additional optical hardware. Results showed significant enhancement in contrast and resolution, showcasing the potential of GAN‐based models for fibre bundle‐based fluorescence microscopy.

show abstract

Deep learning 2D and 3D optical sectioning microscopy using cross-modality Pix2Pix cGAN image translation

Cited by 9 publications

References 26 publications

Exceeding the limit for microscopic image translation with a deep learning-based unified framework

Exceeding the limit for microscopic image translation with a deep learning-based unified framework

Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusion

Enhancing resolution and contrast in fibre bundle‐based fluorescence microscopy using generative adversarial network

Contact Info

Product

Resources

About