DRIT++: Diverse Image-to-Image Translation via Disentangled Representations

Abstract

Image-to-image translation aims to learn the mapping between two visual domains. There are two main challenges for this task: 1) lack of aligned training pairs and 2) multiple possible outputs from a single input image. In this work, we present an approach based on disentangled representation for generating diverse outputs without paired training images. To synthesize diverse outputs, we propose to embed images onto two spaces: a domain-invariant content space capturing shared information across domains and a domainspecific attribute space. Our model takes the encoded content features extracted from a given input and attribute vectors sampled from the attribute space to synthesize diverse outputs at test time. To handle unpaired training data, we introduce a cross-cycle consistency loss based on disentangled representations. Qualitative results show that our model can generate diverse and realistic images on a wide range of tasks without paired training data. For quantitative evaluations, we

ICAM-reg: Interpretable Classification and Regression with Feature Attribution for Mapping Neurological Phenotypes in Individual Scans

“…Over recent years, several deep generative approaches to image-to-image translation have emerged [31], [32], [34]- [37], where these have been applied to many different domains, including medical imaging [30], [38]- [40]. Of these, Lee et al [34], in particular, developed a domain translation network called DRIT (Fig.…”

“…Over recent years, several deep generative approaches to image-to-image translation have emerged [31], [32], [34]- [37], where these have been applied to many different domains, including medical imaging [30], [38]- [40]. Of these, Lee et al [34], in particular, developed a domain translation network called DRIT (Fig. 2b), which constrains translation only to features specific to a class, by encoding separate class-relevant (attribute) and class-irrelevant (content) latent spaces, and employing a discriminator.…”

“…Accordingly, in our work ICAM [33], we extended upon the intuitions of these models to create one framework which allows simultaneous classification and feature attribution, using a more interpretable model. Compared to VA-GAN and DRIT++ [30], [34], ICAM uses 2 shared disentangled latent spaces, attribute and content, which encode for class-relevant and classirrelevant information, respectively. The use of a shared attribute (class) latent space allows the addition of a classification layer (and in this work, also a regression layer) to the network (Fig.…”

“…In addition, the rejection sampling module (Fig. 4) checks the class of a randomly sampled attribute latent vector (using the classification layer) to enable feature detection for a single subject during test time, as well as the analysis of the model's mean and variance by sampling the attribute latent space multiple times (in comparison to DRIT++ [34]). Other components of ICAM such as a FA map loss, L2 reconstruction loss, and a 3D attribute latent space also improve performance compared to VA-GAN and DRIT++ (as illustrated using ablation studies in [33]).…”

“…To address these problems in [33] we developed ICAM (Interpretable Classification via disentangled representations and feature Attribution Mapping); this improved on the stateof-the-art FA methods (Table II) [18], [21], [22], [24], [30] by building on approaches for image-to-image translation [34] to perform feature attribution by disentangling classrelevant attributes (attr) from class-irrelevant content. Sharp reconstructions are then learnt through use of a Variational Autoencoder (VAE) with a discriminator loss on the decoder (Generative Adversarial Network, GAN).…”