Improving Generation and Evaluation of Visual Stories via Semantic Consistency

Maharana, Adyasha; Hannan, Darryl; Bansal, Mohit

doi:10.48550/arxiv.2105.10026

Cited by 5 publications

(16 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are limited number of methods working on the same story visualization task as ours: StoryGAN [14], CP-CSV [24], DUCO [17], and VLC [16]. StoryGAN is based on generative adversarial networks, and CP-CSV, DUCO, and VLC are built on top of StoryGAN, where CP-CSV relies on character segmentation masks to provide an additional supervision, and DUCO and VLC utilized auxiliary captioning networks to keep consistency.…”

Section: Methodsmentioning

confidence: 99%

“…CP-CSV [24] was built on StoryGAN, and utilized segmentation masks to improve the character consistency. DUCO [17] and VLC [16] also adopted StoryGAN as the backbone, and added auxiliary captioning networks to build a text-image-text circle. However, all these methods still struggle with the quality of output images, and fail to generate fine-grained regional details.…”

Section: Related Workmentioning

confidence: 99%

“…Component Analysis on Pororo-SV. "Ours w/o New Sentence Representation" stands for without using the proposed new sentence representation; "Ours w/ Discrimiantor" stands for using the discriminator in current story visualization methods [14,24,17]; "Ours w/o Extended Spatial Attention" stands for without adopting the proposed attention; "Ours w/ Word-Level Spatial Attention" is with the implementation of word-level spatial attention [26], instead of our proposed extended spatial attention. of word-level information via the proposed attention in the generation process, there is no corresponding word-level feedback from the discriminator, regarding whether word-related attributes are produced in synthetic story images, and…”

Section: Component Analysismentioning

confidence: 99%

“…StoryGAN [14] first proposed a sequential conditional GAN-based framework. Using StoryGAN as the backbone, CP-CSV [24] utilized segmentation masks to keep character consistency, and DUCO [17] and VLC [16] adopted additional auxiliary captioning networks to improve text-image semantic alignment. However, all these works still struggle with the quality of output images, and may fail to generate fine-grained image regional details corresponding to different semantic words (see Fig.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Word-Level Fine-Grained Story Visualization

Li¹,

Lukasiewicz²

2022

Preprint

View full text Add to dashboard Cite

Story visualization aims to generate a sequence of images to narrate each sentence in a multi-sentence story with a global consistency across dynamic scenes and characters. Current works still struggle with output images' quality and consistency, and rely on additional semantic information or auxiliary captioning networks. To address these challenges, we first introduce a new sentence representation, which incorporates word information from all story sentences to mitigate the inconsistency problem. Then, we propose a new discriminator with fusion features and further extend the spatial attention to improve image quality and story consistency. Extensive experiments on different datasets and human evaluation demonstrate the superior performance of our approach, compared to state-of-the-art methods, neither using segmentation masks nor auxiliary captioning networks.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Component Analysismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Word-Level Fine-Grained Story Visualization

Li¹,

Lukasiewicz²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This is different from video since it lacks continuous frame prediction, having a temporal relation to show a smooth motion transition. However, the literature reports only a few studies on the story-generation task, most of which are retrieval-based [301][302][303], while some pay attention to GAN models [31,231,233,235,237,239] and almost none of the other generative models are explored, except [230].…”

Section: Story (Consistent)mentioning

confidence: 99%

A Review of Multi-Modal Learning from the Text-Guided Visual Processing Viewpoint

Ullah

Lee

et al. 2022

Sensors

View full text Add to dashboard Cite

For decades, co-relating different data domains to attain the maximum potential of machines has driven research, especially in neural networks. Similarly, text and visual data (images and videos) are two distinct data domains with extensive research in the past. Recently, using natural language to process 2D or 3D images and videos with the immense power of neural nets has witnessed a promising future. Despite the diverse range of remarkable work in this field, notably in the past few years, rapid improvements have also solved future challenges for researchers. Moreover, the connection between these two domains is mainly subjected to GAN, thus limiting the horizons of this field. This review analyzes Text-to-Image (T2I) synthesis as a broader picture, Text-guided Visual-output (T2Vo), with the primary goal being to highlight the gaps by proposing a more comprehensive taxonomy. We broadly categorize text-guided visual output into three main divisions and meaningful subdivisions by critically examining an extensive body of literature from top-tier computer vision venues and closely related fields, such as machine learning and human–computer interaction, aiming at state-of-the-art models with a comparative analysis. This study successively follows previous surveys on T2I, adding value by analogously evaluating the diverse range of existing methods, including different generative models, several types of visual output, critical examination of various approaches, and highlighting the shortcomings, suggesting the future direction of research.

show abstract

A survey on knowledge-enhanced multimodal learning

Lymperaiou,

Stamou

2024

Artif Intell Rev

View full text Add to dashboard Cite

Multimodal learning has been a field of increasing interest, aiming to combine various modalities in a single joint representation. Especially in the area of visiolinguistic (VL) learning multiple models and techniques have been developed, targeting a variety of tasks that involve images and text. VL models have reached unprecedented performances by extending the idea of Transformers, so that both modalities can learn from each other. Massive pre-training procedures enable VL models to acquire a certain level of real-world understanding, although many gaps can be identified: the limited comprehension of commonsense, factual, temporal and other everyday knowledge aspects questions the extendability of VL tasks. Knowledge graphs and other knowledge sources can fill those gaps by explicitly providing missing information, unlocking novel capabilities of VL models. At the same time, knowledge graphs enhance explainability, fairness and validity of decision making, issues of outermost importance for such complex implementations. The current survey aims to unify the fields of VL representation learning and knowledge graphs, and provides a taxonomy and analysis of knowledge-enhanced VL models.

show abstract

Improving Generation and Evaluation of Visual Stories via Semantic Consistency

Cited by 5 publications

References 0 publications

Word-Level Fine-Grained Story Visualization

Word-Level Fine-Grained Story Visualization

A Review of Multi-Modal Learning from the Text-Guided Visual Processing Viewpoint

A survey on knowledge-enhanced multimodal learning

Contact Info

Product

Resources

About