Contrastive Learning for Unpaired Image-to-Image Translation

Park, Taesung; Efros, Alexei A.; Zhang, Richard; Zhu, Jun

doi:10.1007/978-3-030-58545-7_19

Cited by 929 publications

(1,005 citation statements)

References 66 publications

Supporting

Mentioning

1,001

Contrasting

Unclassified

Order By: Relevance

“…Generative Adversarial Networks (GANs) [ 41 ] have been successfully deployed for generating realistic images; in particular, Pix2Pix [ 42 ], CycleGAN [ 43 ], and CUT [ 44 ] have been shown to produce promising results in translating images from one domain to another. Zhang et al introduced a Pix2Pix-based approach that focused on achieving a high face recognition accuracy of their generated visible images by incorporating an explicit closed-set face recognition loss [ 45 ].…”

Section: Resultsmentioning

confidence: 99%

SpeakingFaces: A Large-Scale Multimodal Dataset of Voice Commands with Visual and Thermal Video Streams

Abdrakhmanova

Kuzdeuov

Jarju

et al. 2021

Sensors

View full text Add to dashboard Cite

We present SpeakingFaces as a publicly-available large-scale multimodal dataset developed to support machine learning research in contexts that utilize a combination of thermal, visual, and audio data streams; examples include human–computer interaction, biometric authentication, recognition systems, domain transfer, and speech recognition. SpeakingFaces is comprised of aligned high-resolution thermal and visual spectra image streams of fully-framed faces synchronized with audio recordings of each subject speaking approximately 100 imperative phrases. Data were collected from 142 subjects, yielding over 13,000 instances of synchronized data (∼3.8 TB). For technical validation, we demonstrate two baseline examples. The first baseline shows classification by gender, utilizing different combinations of the three data streams in both clean and noisy environments. The second example consists of thermal-to-visual facial image translation, as an instance of domain transfer.

show abstract

Section: Resultsmentioning

confidence: 99%

SpeakingFaces: A Large-Scale Multimodal Dataset of Voice Commands with Visual and Thermal Video Streams

Abdrakhmanova

Kuzdeuov

Jarju

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Based on that assumption, they propose the geometry consistency to achieve one-sided image translation. CUT [16] introduces contrastive strategy on patch-level to preserve image contents in place of cycle consistency. However, these methods still consider stylization as a global task as they focus on migrating styles or attributes onto the entire images.…”

Section: Image Translation Without Cycle Consistencymentioning

confidence: 99%

Label-Free Regional Consistency for Image-to-Image Translation

Guo

Zhou

Zhou³

et al. 2021

2021 IEEE International Conference on Multimedia and Expo (ICME)

View full text Add to dashboard Cite

Image-to-Image translation aims to translate images from one domain to another. Existing approaches mainly stylize the images globally, while the local consistency between regions has been under-explored. Some instance-aware methods capture the regional consistency but heavily depend on wellannotated labels of a large-scale dataset. Besides, we observe that content-alike regions should have similar style between the target and translated images, however, little attention has been paid to explore such intrinsic property as explicit prior knowledge to guide the image translation process. In this paper, we aim to explore the label-free regional consistency for image-to-image translation. We propose regional relation consistency not only to maintain the global structure but also to keep a close look at the regional consistency, thus achieving more rigorous preservation of image contents. Moreover, we employ the phase of images as a semantic prior to select regions with similar content. We present phase-guided amplitude consistency to perform a more efficient local stylization. Extensive experiments verify that our approach outperforms the existing methods with a clear margin.

show abstract

“…For the generation we use two metrics that measure slightly different aspects, the Fréchet Inception Distance (FID) and the Structure Similarity Index Measure (SSIM). These have been applied in works of similar context [3,7,11,12,16,18,[24][25][26]. For semantic segmentation purposes we opt for the DICE score.…”

Section: Evaluation Metricsmentioning

confidence: 99%

“…This framework was used for the synthesis of abnormal brain MRI images from annotations with a tumor label [14], and for the generation of retinal images from vessel masks [15]. A more recent work based on Contrastive Learning [16] has tried to both improve and generalize the image synthesis approach by disregarding the bijective assumption implicit in works using cycle-consistency and still avoid tailoring a loss function to the setup.…”

Section: Introductionmentioning

confidence: 99%

Adversarial Data Augmentation on Breast MRI Segmentation

et al. 2021

View full text Add to dashboard Cite

The scarcity of balanced and annotated datasets has been a recurring problem in medical image analysis. Several researchers have tried to fill this gap employing dataset synthesis with adversarial networks (GANs). Breast magnetic resonance imaging (MRI) provides complex, texture-rich medical images, with the same annotation shortage issues, for which, to the best of our knowledge, no previous work tried synthesizing data. Within this context, our work addresses the problem of synthesizing breast MRI images from corresponding annotations and evaluate the impact of this data augmentation strategy on a semantic segmentation task. We explored variations of image-to-image translation using conditional GANs, namely fitting the generator’s architecture with residual blocks and experimenting with cycle consistency approaches. We studied the impact of these changes on visual verisimilarity and how an U-Net segmentation model is affected by the usage of synthetic data. We achieved sufficiently realistic-looking breast MRI images and maintained a stable segmentation score even when completely replacing the dataset with the synthetic set. Our results were promising, especially when concerning to Pix2PixHD and Residual CycleGAN architectures.

show abstract

Contrastive Learning for Unpaired Image-to-Image Translation

Cited by 929 publications

References 66 publications

SpeakingFaces: A Large-Scale Multimodal Dataset of Voice Commands with Visual and Thermal Video Streams

SpeakingFaces: A Large-Scale Multimodal Dataset of Voice Commands with Visual and Thermal Video Streams

Label-Free Regional Consistency for Image-to-Image Translation

Adversarial Data Augmentation on Breast MRI Segmentation

Contact Info

Product

Resources

About