Semantically Invariant Text-to-Image Generation

Sah, Shagan; Peri, Dheeraj; Shringi, Ameya; Zhang, Chi; Domínguez, M.; Savakis, Andreas; Ptucha, Rav

doi:10.1109/icip.2018.8451656

Cited by 10 publications

(10 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As a result, many approaches now use attention mechanisms to attend to specific words of the sentence [7], use intermediate representations such as scene layouts [2], condition on additional information such as object bounding boxes [3] or perform interactive image refinement [12]. Other approaches generate images directly from semantic layouts without additional textual input [13], [14]or perform a translation from text to images and back [15], [16].…”

Section: Related Workmentioning

confidence: 99%

“…captions containing the phrase "hot dog" are evaluated based on the assumption that the image should contain a dog). [15] introduce a detection score that calculates (roughly) whether a pre-trained object detector detects an object in a generated image with high certainty. However, no information from the caption is taken into account, meaning any detection with high confidence is "good" even if the detected object does not make sense in the context of the caption.…”

Section: Semantic Object Accuracy (Soa)mentioning

confidence: 99%

See 1 more Smart Citation

Semantic Object Accuracy for Generative Text-to-Image Synthesis

Hinz

Heinrich

Wermter

2022

IEEE Trans. Pattern Anal. Mach. Intell.

106

View full text Add to dashboard Cite

Generative adversarial networks conditioned on textual image descriptions are capable of generating realistic-looking images. However, current methods still struggle to generate images based on complex image captions from a heterogeneous domain. Furthermore, quantitatively evaluating these text-to-image models is challenging, as most evaluation metrics only judge image quality but not the conformity between the image and its caption. To address these challenges we introduce a new model that explicitly models individual objects within an image and a new evaluation metric called Semantic Object Accuracy (SOA) that specifically evaluates images given an image caption. The SOA uses a pre-trained object detector to evaluate if a generated image contains objects that are mentioned in the image caption, e.g. whether an image generated from "a car driving down the street" contains a car. We perform a user study comparing several text-to-image models and show that our SOA metric ranks the models the same way as humans, whereas other metrics such as the Inception Score do not. Our evaluation also shows that models which explicitly model objects outperform models which only model global image characteristics.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Semantic Object Accuracy (Soa)mentioning

confidence: 99%

Semantic Object Accuracy for Generative Text-to-Image Synthesis

Hinz

Heinrich

Wermter

2022

IEEE Trans. Pattern Anal. Mach. Intell.

106

View full text Add to dashboard Cite

show abstract

“…"hot dog" for "dog") in addition to a list of false positive captions. In contrast to [140] which also proposed a detection based evaluation metric, SOA does take the caption into account. Table A.5 shows SOA scores as reported in [108].…”

Section: Image-text Alignment Metricsmentioning

confidence: 99%

“…Instructions can vary between choosing the "best" without specifying what it means, to precise directives such as rating whether objects are identifiable and/or match the input description. For example, users have been asked to rank images based on the relevance of text [109], to select the image which best depicts the caption [141,108], to rate whether any one object is identifiable, and how well the image aligns with the text given [140], to select the more convincing image and the one which is more semantically consistent with the ground truth [113]. While some report average ranks, others report the ratio of being ranked first.…”

Section: User Studiesmentioning

confidence: 99%

Adversarial Text-to-Image Synthesis: A Review

Frolov,

Hinz,

Raue

et al. 2021

Preprint

View full text Add to dashboard Cite

With the advent of generative adversarial networks, synthesizing images from textual descriptions has recently become an active research area. It is a flexible and intuitive way for conditional image generation with significant progress in the last years regarding visual realism, diversity, and semantic alignment. However, the field still faces several challenges that require further research efforts such as enabling the generation of high-resolution images with multiple objects, and developing suitable and reliable evaluation metrics that correlate with human judgement. In this review, we contextualize the state of the art of adversarial text-to-image synthesis models, their development since their inception five years ago, and propose a taxonomy based on the level of supervision. We critically examine current strategies to evaluate textto-image synthesis models, highlight shortcomings, and identify new areas of research, ranging from the development of better datasets and evaluation metrics to possible improvements in architectural design and model training. This review complements previous surveys on generative adversarial networks with a focus on text-to-image synthesis which we believe will help researchers to further advance the field.

show abstract

“…Given a scene sketch, Gao et al [7] implemented a controllable image generation method to meet the specific requirements. Also, there are many generative models [18,9,28,25] that take the text as the input for multi-modal text-to-image generation.…”

Section: 2mentioning

confidence: 99%

Global-Affine and Local-Specific Generative Adversarial Network for semantic-guided image generation

Zhang¹,

Ni²,

Hou³

et al. 2021

MFC

View full text Add to dashboard Cite

The recent progress in learning image feature representations has opened the way for tasks such as label-to-image or text-to-image synthesis. However, one particular challenge widely observed in existing methods is the difficulty of synthesizing fine-grained textures and small-scale instances. In this paper, we propose a novel Global-Affine and Local-Specific Generative Adversarial Network (GALS-GAN) to explicitly construct global semantic layouts and learn distinct instance-level features. To achieve this, we adopt the graph convolutional network to calculate the instance locations and spatial relationships from scene graphs, which allows our model to obtain the highfidelity semantic layouts. Also, a local-specific generator, where we introduce the feature filtering mechanism to separately learn semantic maps for different categories, is utilized to disentangle and generate specific visual features. Moreover, we especially apply a weight map predictor to better combine the global and local pathways considering the highly complementary between these two generation sub-networks. Extensive experiments on the COCO-Stuff and Visual Genome datasets demonstrate the superior generation performance of our model against previous methods, our approach is more capable of capturing photo-realistic local characteristics and rendering small-sized entities with more details.

show abstract

Semantically Invariant Text-to-Image Generation

Cited by 10 publications

References 23 publications

Semantic Object Accuracy for Generative Text-to-Image Synthesis

Semantic Object Accuracy for Generative Text-to-Image Synthesis

Adversarial Text-to-Image Synthesis: A Review

Global-Affine and Local-Specific Generative Adversarial Network for semantic-guided image generation

Contact Info

Product

Resources

About