Novel View Synthesis on Unpaired Data by Conditional Deformable Variational Auto-Encoder

Yin, Mingyu; Li, Sun; Li, Qingli

doi:10.1007/978-3-030-58604-1_6

Cited by 14 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zhu et al introduced CycleGAN, a method capable of recovering the front face from a single profile postural facial image, even when the source domain does not match the target domain [59]. This approach is based on a conditional variational autoencoder and generative adversarial network (cVAE-GAN) framework, which does not require paired data, making it a versatile method for view translation [60].Shen et al proposed Pairwise-GAN, employing two parallel U-Nets as generators and PatchGAN as a discriminator to synthesize frontal face images [61]. Similarly, Chan et al presented pi-GAN, a method utilizing periodic implicit Generative Adversarial Networks for high-quality 3D-aware image synthesis [62].…”

Section: Pose Manipulationmentioning

confidence: 99%

Enhancing image resolution of confocal fluorescence microscopy with deep learning

Huang

Yao

et al. 2023

PhotoniX

View full text Add to dashboard Cite

Super-resolution optical imaging is crucial to the study of cellular processes. Current super-resolution fluorescence microscopy is restricted by the need of special fluorophores or sophisticated optical systems, or long acquisition and computational times. In this work, we present a deep-learning-based super-resolution technique of confocal microscopy. We devise a two-channel attention network (TCAN), which takes advantage of both spatial representations and frequency contents to learn a more precise mapping from low-resolution images to high-resolution ones. This scheme is robust against changes in the pixel size and the imaging setup, enabling the optimal model to generalize to different fluorescence microscopy modalities unseen in the training set. Our algorithm is validated on diverse biological structures and dual-color confocal images of actin-microtubules, improving the resolution from ~ 230 nm to ~ 110 nm. Last but not least, we demonstrate live-cell super-resolution imaging by revealing the detailed structures and dynamic instability of microtubules.

show abstract

Section: Pose Manipulationmentioning

confidence: 99%

Enhancing image resolution of confocal fluorescence microscopy with deep learning

Huang

Yao

et al. 2023

PhotoniX

View full text Add to dashboard Cite

show abstract

“…Paired data of the source and target views are commonly required for novel view synthesis, thus raising the threshold of data acquisition. To address this challenge, Yin et al presented an unpaired view translation framework that used cVAE‐GAN to decompose the features of source views and control the generation of target views through view condition vectors 29 . Furthermore, Palazzi et al proposed a self‐supervised and semiparametric method (a fusion of an entirely learning‐based generative network and a not learned priori geometric knowledge component) that can generate novel views of a vehicle from a single monocular image 30 .…”

Section: Related Workmentioning

confidence: 99%

PGNet: Projection generative network for sparse‐view reconstruction of projection‐based magnetic particle imaging

Gao

et al. 2022

Medical Physics

View full text Add to dashboard Cite

Background: Magnetic particle imaging (MPI) is a novel tomographic imaging modality that scans the distribution of superparamagnetic iron oxide nanoparticles. However, it is time-consuming to scan multiview two-dimensional (2D) projections for three-dimensional (3D) reconstruction in projection MPI, such as computed tomography (CT). An intuitive idea is to use the sparse-view projections for reconstruction to improve the temporal resolution. Tremendous progress has been made toward addressing the sparse-view problem in CT, because of the availability of large data sets. For the novel tomography of MPI, to the best of our knowledge, studies on the sparse-view problem have not yet been reported. Purpose: The acquisition of multiview projections for 3D MPI imaging is time-consuming. Our goal is to only acquire sparse-view projections for reconstruction to improve the 3D imaging temporal resolution of projection MPI. Methods: We propose to address the sparse-view problem in projection MPI by generating novel projections. The data set we constructed consists of three parts: simulation data set (including 3000 3D data), four phantoms data, and an in vivo mouse data. The simulation data set is used to train and validate the network, and the phantoms and in vivo mouse data are used to test the network. When the number of novel generated projections meets the requirements of filtered back projection, the streaking artifacts will be absent from MPI tomographic imaging. Specifically, we propose a projection generative network (PGNet), that combines an attention mechanism, adversarial training strategy, and a fusion loss function and can generate novel projections based on sparse-view real projections. To the best of our knowledge, we are the first to propose a deep learning method to attempt to overcome the sparse-view problem in projection MPI. Results: We compare our method with several sparse-view methods on phantoms and in vivo mouse data and validate the advantages and effectiveness of 2354

show abstract

“…2D transformation-based methods mainly focus on learning pixel displacement between the input source view(s) and the target view [8]- [10] or directly regressing the pixel colors of the target view in its 2D image plane [11]- [13]. 3D transformation-based methods [14]- [15] often predict a 3D representation, such as an occupancy volume, first and then explicitly perform 3D spatial transformation on the representation to synthesize the target view.…”

Section: Related Workmentioning

confidence: 99%

Novel View Synthesis from a Single Image via Unsupervised learning

Liu¹,

Lei²,

Peng³

et al. 2021

Preprint

View full text Add to dashboard Cite

View synthesis aims to generate novel views from one or more given source views. Although existing methods have achieved promising performance, they usually require paired views of different poses to learn a pixel transformation. This paper proposes an unsupervised network to learn such a pixel transformation from a single source viewpoint. In particular, the network consists of a token transformation module (TTM) that facilities the transformation of the features extracted from a source viewpoint image into an intrinsic representation with respect to a pre-defined reference pose and a view generation module (VGM) that synthesizes an arbitrary view from the representation. The learned transformation allows us to synthesize a novel view from any single source viewpoint image of unknown pose. Experiments on the widely used view synthesis datasets have demonstrated that the proposed network is able to produce comparable results to the state-of-the-art methods despite the fact that learning is unsupervised and only a single source viewpoint image is required for generating a novel view. The code will be available soon.

show abstract

Novel View Synthesis on Unpaired Data by Conditional Deformable Variational Auto-Encoder

Cited by 14 publications

References 25 publications

Enhancing image resolution of confocal fluorescence microscopy with deep learning

Enhancing image resolution of confocal fluorescence microscopy with deep learning

PGNet: Projection generative network for sparse‐view reconstruction of projection‐based magnetic particle imaging

Novel View Synthesis from a Single Image via Unsupervised learning

Contact Info

Product

Resources

About