Understanding Events by Eye and Ear: Agent and Verb Drive Non-anticipatory Eye Movements in Dynamic Scenes

Abstract

As Macnamara (1978) once asked, how can we talk about what we see? We report on a study manipulating realistic dynamic scenes and sentences aiming to understand the interaction between linguistic and visual representations in real-world situations. Specifically, we monitored participants’ eye movements as they watched video clips of everyday scenes while listening to sentences describing these scenes. We manipulated two main variables. The first was the semantic class of the verb in the sentence and the second was the action/motion of the agent in the unfolding event. The sentences employed two verb classes–causatives (e.g., break) and perception/psychological (e.g., notice)–which impose different constraints on the nouns that serve as their grammatical complements. The scenes depicted events in which agents either moved toward a target object (always the referent of the verb-complement noun), away from it, or remained neutral performing a given activity (such as cooking). Scenes and sentences were synchronized such that the verb onset corresponded to the first video frame of the agent motion toward or away from the object. Results show effects of agent motion but weak verb-semantic restrictions: causatives draw more attention to potential referents of their grammatical complements than perception verbs only when the agent moves toward the target object. Crucially, we found no anticipatory verb-driven eye movements toward the target object, contrary to studies using non-naturalistic and static scenes. We propose a model in which linguistic and visual computations in real-world situations occur largely independent of each other during the early moments of perceptual input, but rapidly interact at a central, conceptual system using a common, propositional code. Implications for language use in real world contexts are discussed.

Causal inference: relating language to event representations and events in the world

“…The generalizability and reliability of the expectation-driven account of language processing have been challenged (see the review by Huettig and Mani, 2016 ), especially for bilinguals using their second language (in reading comprehension: Martin et al, 2013 ; in auditory comprehension: Ito et al, 2018 ) and with low-literacy subjects ( Mishra et al, 2012 ). Additionally, verb-driven anticipatory looks to the target object—used in many studies to argue for expectation-driven processing—appeared absent when realistic and dynamic scenes were presented to participants ( Andersson et al, 2011 ; De Almeida et al, 2019 ; but see counterevidence from Heyselaar et al, 2020 ). Nevertheless, for young proficient monolingual language users at universities, there seems to be good evidence in favor of at least partially expectation-driven language processing, which in turn can be viewed as a stepping stone to understanding the timeline of causal inference.…”

Analisis Pengetahuan Pemeliharaan Kesehatan Gigi dan Mulut pada Remaja di Kelurahan Mahawu, Kota Manado

“…Seseorang memperoleh pengetahuan secara umum melalui indera telinga dan mata. Melalui indera pendengaran dan penglihatan, seseorang memperoleh informasi yang tersampaikan secara linguistik dan visual dan terekam di dalam memori membentuk suatu persepsi (de Almeida et al, 2019). Menurut tabel distribusi pengetahuan, responden sebahagian besar berada pada kategori "Baik" sebesar 20 orang (66,7%) dan kategori "Kurang baik" sebanyak 10 orang (33,3%).…”

Analysing data from the psycholinguistic visual-world paradigm: Comparison of different analysis methods

“…Glossary). Additional decisions in setting up the experiment concern other design-related issues (e.g., the assignment of item–condition combinations to lists and counterbalancing), the stimuli and their timing (e.g., the type of visual input, i.e., scene vs Ersatz-scene, see Henderson & Ferreira, 2004 ; timing: static vs dynamic scenes and the scene preview time, see de Almeida et al, 2019 ; speech rate and the spacing of words, see Andersson et al, 2011 ), and choice of interest period (time window for the analysis) (see Huettig et al, 2011 ).…”