Visually Situated Language Comprehension

Knoeferle, Pia; Guerra, Ernesto

doi:10.1111/lnc3.12177

Cited by 26 publications

(18 citation statements)

References 85 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, Connell and Lynott (2012) , assumed that conceptual processing interacts with the perceptual system. Indeed, decades of eye-tracking evidence have revealed that the language system is highly sensitive to a variety of contextual informational sources, from overt referential relations between nouns and objects to subtler mappings between spatial representation and abstract semantics (see, e.g., Knoeferle and Guerra, 2016 for a review). Yet, we believe that existing evidence has not yet shown unequivocally that perceptual representations have a functional role in conceptual processing (but see Amsel et al, 2014 ), and recent results suggest that causal relation might be task-dependent (e.g., Ostarek and Huettig, 2017 ).…”

Section: Discussionmentioning

confidence: 99%

Semantic Interference and Facilitation: Understanding the Integration of Spatial Distance and Conceptual Similarity During Sentence Reading

Guerra

Knoeferle

2018

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

Existing evidence has shown a processing advantage (or facilitation) when representations derived from a non-linguistic context (spatial proximity depicted by gambling cards moving together) match the semantic content of an ensuing sentence. A match, inspired by conceptual metaphors such as ‘similarity is closeness’ would, for instance, involve cards moving closer together and the sentence relates similarity between abstract concepts such as war and battle. However, other studies have reported a disadvantage (or interference) for congruence between the semantic content of a sentence and representations of spatial distance derived from this sort of non-linguistic context. In the present article, we investigate the cognitive mechanisms underlying the interaction between the representations of spatial distance and sentence processing. In two eye-tracking experiments, we tested the predictions of a mechanism that considers the competition, activation, and decay of visually and linguistically derived representations as key aspects in determining the qualitative pattern and time course of that interaction. Critical trials presented two playing cards, each showing a written abstract noun; the cards turned around, obscuring the nouns, and moved either farther apart or closer together. Participants then read a sentence expressing either semantic similarity or difference between these two nouns. When instructed to attend to the nouns on the cards (Experiment 1), participants’ total reading times revealed interference between spatial distance (e.g., closeness) and semantic relations (similarity) as soon as the sentence explicitly conveyed similarity. But when instructed to attend to the cards (Experiment 2), cards approaching (vs. moving apart) elicited first interference (when similarity was implicit) and then facilitation (when similarity was made explicit) during sentence reading. We discuss these findings in the context of a competition mechanism of interference and facilitation effects.

show abstract

Section: Discussionmentioning

confidence: 99%

Semantic Interference and Facilitation: Understanding the Integration of Spatial Distance and Conceptual Similarity During Sentence Reading

Guerra

Knoeferle

2018

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Over the past few decades, research using online behavioral measures such as priming, self‐paced reading, and eye‐tracking has helped paint a picture of a language processing system that is highly interactive, continuously incorporating input from a variety of contextual sources (cf. Knoeferle & Guerra, ; Marslen‐Wilson, ; McRae & Matsuki, ; Onnis & Spivey, ). The utility of these methods derives from the temporal nature of language: As language unfolds, it is filled with temporary phonetic, lexical, syntactic, and semantic ambiguities.…”

Section: Behavioral Measures Of Interactive Language Processing In a mentioning

confidence: 99%

Interactivity of language

Spevack¹,

Falandays²,

Batzloff³

et al. 2018

Language and Linguist. Compass

View full text Add to dashboard Cite

In previous decades, the language sciences made important advances by dividing language into its different information formats, such as phonetics, semantics, and syntax. Such division generally implied that language processing is divorced from context. In more recent decades, however, important advances in the language sciences have been made in understanding how linguistic information interacts with context. These contextual influences stem from a broad range of sources. They include linguistic and non‐linguistic processes within and between individuals. This brief review touches on experimental results from both behavioral and neural measures, and from both individuals processing prepared linguistic stimuli and dyads sharing unscripted conversation. Overall, the findings generally support a view of language processing that must somehow allow for the different information formats of language to retain their unique labels but also accommodate the fact that they frequently interact and overlap with other, even non‐linguistic, formats of information.

show abstract

“…Eye tracking has, however, also been applied to examine spoken language comprehension during the inspection of objects in the world (or on a computer display . Such eye gaze data have also shed light on a range of further comprehension processes, such as lexical ambiguity resolution 10,11 , pronoun resolution 12 , the disambiguation of structural and thematic role assignment by means of information in the visual context 13,14,16 , and pragmatic processes 15 , among many others 4 . Clearly, the eye movements to objects during language comprehension can be informative of the implicated processes.…”

Section: Introductionmentioning

confidence: 99%

Eye Tracking During Visually Situated Language Comprehension: Flexibility and Limitations in Uncovering Visual Context Effects

Ronderos

Münster

Guerra

et al. 2018

JoVE

Self Cite

View full text Add to dashboard Cite

The present work is a description and an assessment of a methodology designed to quantify different aspects of the interaction between language processing and the perception of the visual world. The recording of eye-gaze patterns has provided good evidence for the contribution of both the visual context and linguistic/world knowledge to language comprehension. Initial research assessed object-context effects to test theories of modularity in language processing. In the introduction, we describe how subsequent investigations have taken the role of the wider visual context in language processing as a research topic in its own right, asking questions such as how our visual perception of events and of speakers contributes to comprehension informed by comprehenders' experience. Among the examined aspects of the visual context are actions, events, a speaker's gaze, and emotional facial expressions, as well as spatial object configurations. Following an overview of the eyetracking method and its different applications, we list the key steps of the methodology in the protocol, illustrating how to successfully use it to study visually-situated language comprehension. A final section presents three sets of representative results and illustrates the benefits and limitations of eye tracking for investigating the interplay between the perception of the visual world and language comprehension. Video Link The video component of this article can be found at https://www.jove.com/video/57694/ 2). In this 'visual world' eye-tracking version, the inspection of objects is guided by language. When the comprehenders hear the zebra, for instance, their inspection of a zebra on the screen is taken to reflect that they are thinking about the animal. In what is known as the visual world paradigm, a comprehender's eye gaze is taken to reflect spoken language comprehension and the activation of associated knowledge (e.g., listeners also inspect the zebra when they hear grazing, indicating an action performed by the zebra) 2. Such inspections suggest a systematic link between language-world relations and eye movements 2. A common way to quantify this link is by computing the proportion of looks to different predetermined regions on a screen. This allows researchers to directly compare (across conditions, by participants and items) the amount of attention given to different objects at a particular time and how these values change in millisecond resolution.

show abstract

Visually Situated Language Comprehension

Cited by 26 publications

References 85 publications

Semantic Interference and Facilitation: Understanding the Integration of Spatial Distance and Conceptual Similarity During Sentence Reading

Semantic Interference and Facilitation: Understanding the Integration of Spatial Distance and Conceptual Similarity During Sentence Reading

Interactivity of language

Eye Tracking During Visually Situated Language Comprehension: Flexibility and Limitations in Uncovering Visual Context Effects

Contact Info

Product

Resources

About