ChefGAN

Abstract

Although significant progress has been made in generating images from the text by using generative adversarial networks (GANs), it is still challenging to deal with long text, which contains complex semantic information like recipes. This paper focuses on generating images with high visual realism and semantic consistency from the complex text of recipes. To achieve this, we propose a GANs based method termed ChefGAN. The critical concept of ChefGAN is that a joint image-recipe embedding model is used before the generation task to provide high-quality representations of recipes, and it acts as an extra regularization during the generation to improve semantic consistency. Two modules are designed for this image text embedding module (ITEM) and a cascaded image generation module (CIGM). The generation process is carried out in 3 steps: (1) Two encoders in ITEM are trained simultaneously to generate similar representations for each image-recipe pair. (2) CIGM generates images according to the representations from ITEM's text encoder. (3) The generated image is fed into ITEM's image encoder to calculate the similarity with the given recipe. This process can provide additional regularization effect other than the impact of a discriminator. To facilitate convergence, we applied a two-stage training strategy, which generates an image with low resolution and then one with high resolution in the CIGM module. Compared with other representative state-of-the-art methods, ChefGAN demonstrates better performance both in visual realism and semantic consistency. CCS CONCEPTS • Information systems → Multimedia content creation; • Computing methodologies → Computer vision representations.

Design Complexity: Assessing Cross-Modal Correspondence between Complex Food Images and the Desire to Eat.

“…Complex images have high visual realism and semantic consistency; the bandwidth resolution is high (design complexity) and consists of various degrees of elements in the layout (feature complexity) intended to evoke the processing of the cues presented. Simple images with low bandwidth resolution without multiple components are designed to dissuade the processing capabilities of the cue given (Da Silva et al, 2011;Pan et al, 2020;Yu & Winkler, 2013). number of elements present in any given layout, and design complexity accounts for the detailed variation in the layout that conveys the primary visual form, such as color, shape, brightness, and edge patterns of a layout.…”

CREAMY: Cross-Modal Recipe Retrieval By Avoiding Matching Imperfectly

“…Wang et al [62] introduced a cycle-consistency training method, which improved image generation by optimizing the inverted latent codes. Chef-GAN [63] involved a joint image-recipe embedding model to GANs before and during the stage of generate images. CookGAN [64] mimicked visual effect of instructions and preserved the fine-grained details of images.…”