Describing Events: Changes in Eye Movements and Language Production Due to Visual and Conceptual Properties of Scenes

Abstract

How can a visual environment shape our utterances? A variety of visual and conceptual factors appear to affect sentence production, such as the visual cueing of patients or agents, their position relative to one another, and their animacy. These factors have previously been studied in isolation, leaving the question about their interplay open. The present study brings them together to examine systematic variations in eye movements, speech initiation and voice selection in descriptions of visual scenes. A sample of 44 native speakers of German were asked to describe depicted event scenes presented on a computer screen, while both their utterances and eye movements were recorded. Participants were instructed to produce one-sentence descriptions. The pictures depicted scenes with animate agents and either animate or inanimate patients who were situated to the right or to the left of agents. Half of the patients were preceded by a visual cue – a small circle appearing for 60 ms on a blank screen in the place of patients. The results show that scenes with left- rather than right-positioned patients lead to longer speech onset times, a higher probability of passive sentences and looks toward the patient. In addition, scenes with animate patients received more looks and elicited more passive utterances than scenes with inanimate patients. Visual cueing did not produce significant changes in speech, even though there were more looks to cued vs. non-cued referents, demonstrating that cueing only impacted initial scene scanning patterns but not speech. Our findings demonstrate that when examined together rather than separately, visual and conceptual factors of event scenes influence different aspects of behavior. In comparison to cueing that only affected eye movements, patient animacy also acted on the syntactic realization of utterances, whereas patient position in addition altered their onset. In terms of time course, visual influences are rather short-lived, while conceptual factors have long-lasting effects.

Referent Cueing, Position, and Animacy as Accessibility Factors in Visually Situated Sentence Production

“…In particular, attentional cueing seems to exert a comparatively strong effect in speakers of English, as reflected by the high rate of passives produced in this condition (around 20-30% of passives, see e.g., Myachykov et al, 2018, also see Gleitman et al, 2007;Myachykov et al, 2011Myachykov et al, , 2012a. By contrast, attentional cueing does not seem to be equally effective in languages that make use of a richer set of inflectional morphology (i.e., case markings on nouns, Esaulova et al, 2019 for German, also see Hwang and Kaiser, 2015 for Korean). For instance, unlike speakers of English, German speakers did not produce significantly more passives when their attention was cued by a visual stimulus, despite the fact that cueing influenced German speakers' looking behavior (Esaulova et al, 2019).…”

“…By contrast, attentional cueing does not seem to be equally effective in languages that make use of a richer set of inflectional morphology (i.e., case markings on nouns, Esaulova et al, 2019 for German, also see Hwang and Kaiser, 2015 for Korean). For instance, unlike speakers of English, German speakers did not produce significantly more passives when their attention was cued by a visual stimulus, despite the fact that cueing influenced German speakers' looking behavior (Esaulova et al, 2019). When the attentional cue was presented for a longer duration, German speakers' rate of passives increased to around 11% but still did not reach the high proportion of passives produced by Englishspeaking participants (Esaulova, Dolscheid, and Penke, n.d.).…”

“…On half of the trials, the patient was located on the right of the agent, whereas on the other half the patient was located on the left of the agent. The results revealed that participants were faster in initiating their event descriptions when the patient appeared on the right, rather than on the left of the agent (Esaulova et al, 2019). Participants were also more inclined to produce passives when the patient character was located on the left than on the right, demonstrating that the position of referents exerts a substantial effect on speakers' structural choices (for a recent replication in speakers of Russian, see Pokhoday et al, 2019).…”

“…However, while Myachykov and colleagues carefully distinguished between effects of accessibility and visual attention, yet another factor seems to play an important role regarding visually elicited sentence production. In particular, there is recent evidence that the visuospatial position of a referent (i.e., whether a patient is depicted to the right or to the left of an agent) can affect speakers' structural choices (Esaulova et al, 2019). Esaulova et al (2019) examined the effect of position in a sentence production experiment using eye-tracking.…”

“…In particular, there is recent evidence that the visuospatial position of a referent (i.e., whether a patient is depicted to the right or to the left of an agent) can affect speakers' structural choices (Esaulova et al, 2019). Esaulova et al (2019) examined the effect of position in a sentence production experiment using eye-tracking. German-speaking participants were asked to describe scenes depicting simple transitive events with two characters (e.g., a fisherman filming a clown).…”