Unsupervised Image-to-Image Translation: A Review

Hoyez, Henri; Schockaert, Cédric; Rambach, Jason; Mirbach, Bruno; Stricker, Didier

doi:10.3390/s22218540

Cited by 23 publications

(5 citation statements)

References 108 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, our experience indicates that training CycleGAN is not always easy to converge, especially when the image resolution is coarse with a complex background. Nevertheless, it is possible to replace CycleGAN with other SOTA domain adaptation algorithms [ 48 ].…”

Section: Discussionmentioning

confidence: 99%

Enhancing Green Fraction Estimation in Rice and Wheat Crops: A Self-Supervised Deep Learning Semantic Segmentation Approach

gao

Rui-bo

et al. 2023

Plant Phenomics

View full text Add to dashboard Cite

The green fraction (GF), which is the fraction of green vegetation in a given viewing direction, is closely related to the light interception ability of the crop canopy. Monitoring the dynamics of GF is therefore of great interest for breeders to identify genotypes with high radiation use efficiency. The accuracy of GF estimation depends heavily on the quality of the segmentation dataset and the accuracy of the image segmentation method. To enhance segmentation accuracy while reducing annotation costs, we developed a self-supervised strategy for deep learning semantic segmentation of rice and wheat field images with very contrasting field backgrounds. First, the Digital Plant Phenotyping Platform was used to generate large, perfectly labeled simulated field images for wheat and rice crops, considering diverse canopy structures and a wide range of environmental conditions (sim dataset). We then used the domain adaptation model cycle-consistent generative adversarial network (CycleGAN) to bridge the reality gap between the simulated and real images (real dataset), producing simulation-to-reality images (sim2real dataset). Finally, 3 different semantic segmentation models (U-Net, DeepLabV3+, and SegFormer) were trained using 3 datasets (real, sim, and sim2real datasets). The performance of the 9 training strategies was assessed using real images captured from various sites. The results showed that SegFormer trained using the sim2real dataset achieved the best segmentation performance for both rice and wheat crops (rice: Accuracy = 0.940, F1-score = 0.937; wheat: Accuracy = 0.952, F1-score = 0.935). Likewise, favorable GF estimation results were obtained using the above strategy (rice: R 2 = 0.967, RMSE = 0.048; wheat: R 2 = 0.984, RMSE = 0.028). Compared with SegFormer trained using a real dataset, the optimal strategy demonstrated greater superiority for wheat images than for rice images. This discrepancy can be partially attributed to the differences in the backgrounds of the rice and wheat fields. The uncertainty analysis indicated that our strategy could be disrupted by the inhomogeneity of pixel brightness and the presence of senescent elements in the images. In summary, our self-supervised strategy addresses the issues of high cost and uncertain annotation accuracy during dataset creation, ultimately enhancing GF estimation accuracy for rice and wheat field images. The best weights we trained in wheat and rice are available: https://github.com/PheniX-Lab/sim2real-seg .

show abstract

Section: Discussionmentioning

confidence: 99%

Enhancing Green Fraction Estimation in Rice and Wheat Crops: A Self-Supervised Deep Learning Semantic Segmentation Approach

gao

Rui-bo

et al. 2023

Plant Phenomics

View full text Add to dashboard Cite

show abstract

“…Network Architecture: For the asymmetric image transformation, the concept of cycle-consistent loss takes precedence (e.g., CycleGAN, [25] UNIT, MUNIT, and MuGAN [48][49][50] ). These approaches involved learning bidirectional mapping between the input and target domains.…”

Section: Methodsmentioning

confidence: 99%

Noninvasive Nonlinear Optical Computational Histology

Shen,

Li,

Pan

et al. 2023

Advanced Science

View full text Add to dashboard Cite

Cancer remains a global health challenge, demanding early detection and accurate diagnosis for improved patient outcomes. An intelligent paradigm is introduced that elevates label‐free nonlinear optical imaging with contrastive patch‐wise learning, yielding stain‐free nonlinear optical computational histology (NOCH). NOCH enables swift, precise diagnostic analysis of fresh tissues, reducing patient anxiety and healthcare costs. Nonlinear modalities are evaluated, including stimulated Raman scattering and multiphoton imaging, for their ability to enhance tumor microenvironment sensitivity, pathological analysis, and cancer examination. Quantitative analysis confirmed that NOCH images accurately reproduce nuclear morphometric features across different cancer stages. Key diagnostic features, such as nuclear morphology, size, and nuclear‐cytoplasmic contrast, are well preserved. NOCH models also demonstrate promising generalization when applied to other pathological tissues. The study unites label‐free nonlinear optical imaging with histopathology using contrastive learning to establish stain‐free computational histology. NOCH provides a rapid, non‐invasive, and precise approach to surgical pathology, holding immense potential for revolutionizing cancer diagnosis and surgical interventions.

show abstract

“…CycleGAN introduces adversarial and cycle consistency losses to maintain image set characteristics across domains as visualized in Figure 2. Unlike Pix2Pix, it uses an autoencoder structure, lacks skip connections, and doesn't employ a conditional GAN [6], [9], [12].…”

Section: Relevant Workmentioning

confidence: 99%

“…These GANs can be used for various goals. These goals entail domain adaptation, which transforms styles from an image to a different domain style, or style transfer, which keeps most of the input content while changing its style aspects [5], [8], [9]. However, current I2I GAN models still pose problems of mode collapse, instability, and lack of diversity [1].…”

Section: Introductionmentioning

confidence: 99%

Reviewing Fid and Sid Metrics on Generative Adversarial Networks

Deijn,

Batra,

Koch

et al. 2024

AI, Machine Learning and Applications

View full text Add to dashboard Cite

The growth of generative adversarial network (GAN) models has increased the ability of image processing and provides numerous industries with the technology to produce realistic image transformations. However, with the field being recently established there are new evaluation metrics that can further this research. Previous research has shown the Fréchet Inception Distance (FID) to be an effective metric when testing these image-to-image GANs in real-world applications. Signed Inception Distance (SID), a founded metric in 2023, expands on FID by allowing unsigned distances. This paper usespublic datasetsthat consist of façades, cityscapes, and maps within Pix2Pix and CycleGAN models. After trainin , these models are evaluated on both inception distance metrics which measure the generating performance of the trained models. Our findings indicate that usage of the metric SID incorporates an efficient and effective metric to complement, or even exceed the ability shown using the FID for the image-to-image GANs.

show abstract

Unsupervised Image-to-Image Translation: A Review

Cited by 23 publications

References 108 publications

Enhancing Green Fraction Estimation in Rice and Wheat Crops: A Self-Supervised Deep Learning Semantic Segmentation Approach

Enhancing Green Fraction Estimation in Rice and Wheat Crops: A Self-Supervised Deep Learning Semantic Segmentation Approach

Noninvasive Nonlinear Optical Computational Histology

Reviewing Fid and Sid Metrics on Generative Adversarial Networks

Contact Info

Product

Resources

About