The Interplay of Words and Images in Expressing Multimodal Metaphors in Comics

Tasić, Miloš; Stamenković, Dušan

doi:10.1016/j.sbspro.2015.11.308

Cited by 23 publications

(10 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…As in sentences, units in a visual narrative can involve long‐distance connections between nonjuxtaposed images, including center‐embedded “clauses” (Cohn, ). Some sequences may be ambiguous, where a single structural sequence has multiple interpretations and/or parsings (Cohn, , ), or may depict complex semantic relations like metaphor, which may not be motivated by an event structure (Cohn, ; Tasić & Stamenković, ). In addition, the same general meaning can be expressed in multiple different ways that vary what is shown when and how (Brewer & Lichtenstein, ; Cohn, , ; McCloud, )—warranting a system separate from meaning to allow such differences in presentation.…”

Section: Narrative Processingmentioning

confidence: 99%

Your Brain on Comics: A Cognitive Model of Visual Narrative Comprehension

Cohn¹

2019

Topics in Cognitive Science

View full text Add to dashboard Cite

The past decade has seen a rapid growth of cognitive and brain research focused on visual narratives like comics and picture stories. This paper will summarize and integrate this emerging literature into the Parallel Interfacing Narrative‐Semantics Model (PINS Model)—a theory of sequential image processing characterized by an interaction between two representational levels: semantics and narrative structure. Ongoing semantic processes build meaning into an evolving mental model of a visual discourse. Updating of spatial, referential, and event information then incurs costs when they are discontinuous with the growing context. In parallel, a narrative structure organizes semantic information into coherent sequences by assigning images to categorical roles, which are then embedded within a hierarchic constituent structure. Narrative constructional schemas allow for specific predictions of structural sequencing, independent of semantics. Together, these interacting levels of representation engage in an iterative process of retrieval of semantic and narrative information, prediction of upcoming information based on those assessments, and subsequent updating based on discontinuity. These core mechanisms are argued to be domain‐general—spanning across expressive systems—as suggested by similar electrophysiological brain responses (N400, P600, anterior negativities) generated in response to manipulation of sequential images, music, and language. Such similarities between visual narratives and other domains thus pose fundamental questions for the linguistic and cognitive sciences.

show abstract

Section: Narrative Processingmentioning

confidence: 99%

Your Brain on Comics: A Cognitive Model of Visual Narrative Comprehension

Cohn¹

2019

Topics in Cognitive Science

View full text Add to dashboard Cite

show abstract

“…Rather, these images are like a visual list (here, about different parts of a decrepit house), and would remain coherent as a sequence even if the images were rearranged (unlike Figure 1(a/c)). These sequential properties of multimodal relations go unaccounted for in descriptions of surface semantic relations alone (McCloud 1993;Stainbrook 2016;Bateman andWildfeuer 2014a, 2014b;Tasić and Stamenković 2015): namely, where verbally weighted semantics differ in their contributions of narrative structure.…”

Section: Introductionmentioning

confidence: 99%

A Picture is Worth More Words Over Time: Multimodality and Narrative Structure Across Eight Decades of American Superhero Comics

Cohn

Taylor

Pederson

2017

Multimodal Communication

View full text Add to dashboard Cite

The visual narratives of comics involve complex multimodal interactions between written language and the visual language of images, where one or the other may guide the meaning and/or narrative structure. We investigated this interaction in a corpus analysis across eight decades of American superhero comics (1940–2010s). No change across publication date was found for multimodal interactions that weighted meaning towards text or across both text and images, where narrative structures were present across images. However, we found an increase over time of narrative sequences with meaning weighted to the visuals, and an increase of sequences without text at all. These changes coincided with an overall reduction in the number of words per panel, a shift towards panel framing with single characters and close-ups rather than whole scenes, and an increase in shifts between temporal states between panels. These findings suggest that storytelling has shifted towards investing more information in the images, along with an increasing complexity and maturity of the visual narrative structures. This has shifted American comics from being textual stories with illustrations to being visual narratives that use text.

show abstract

“…There are a variety of ways in which texts and images are combined in multimodal content (Hendricks et al, 2016;. Based on our review of the literature and observation of the samples in our dataset, we follow Tasić and Stamenković (2015) and divide multimodal metaphor into three categories: text dominant, image dominant, and complementary. Sometimes metaphors are expressed through texts with a mapping between source and target domains while the accompanying images serve as a visual illustration of the metaphors in the text, which is text dominant.…”

Section: Metaphor Annotationmentioning

confidence: 99%

MultiMET: A Multimodal Dataset for Metaphor Understanding

Zhang¹,

Zhang²,

Zhang³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Metaphor involves not only a linguistic phenomenon, but also a cognitive phenomenon structuring human thought, which makes understanding it challenging. As a means of cognition, metaphor is rendered by more than texts alone, and multimodal information in which vision/audio content is integrated with the text can play an important role in expressing and understanding metaphor. However, previous metaphor processing and understanding has focused on texts, partly due to the unavailability of large-scale datasets with ground truth labels of multimodal metaphor. In this paper, we introduce MultiMET, a novel multimodal metaphor dataset to facilitate understanding metaphorical information from multimodal text and image. It contains 10,437 text-image pairs from a range of sources with multimodal annotations of the occurrence of metaphors, domain relations, sentiments metaphors convey, and author intents. MultiMET opens the door to automatic metaphor understanding by investigating multimodal cues and their interplay. Moreover, we propose a range of strong baselines and show the importance of combining multimodal cues for metaphor understanding. MultiMET will be released publicly for research.

show abstract

The Interplay of Words and Images in Expressing Multimodal Metaphors in Comics

Cited by 23 publications

References 3 publications

Your Brain on Comics: A Cognitive Model of Visual Narrative Comprehension

Your Brain on Comics: A Cognitive Model of Visual Narrative Comprehension

A Picture is Worth More Words Over Time: Multimodality and Narrative Structure Across Eight Decades of American Superhero Comics

MultiMET: A Multimodal Dataset for Metaphor Understanding

Contact Info

Product

Resources

About