On the temporal dynamics of language-mediated vision and vision-mediated language

Abstract

Recent converging evidence suggests that language and vision interact immediately in non-trivial ways, although the exact nature of this interaction is still unclear. Not only does linguistic information influence visual perception in real-time, but visual information also influences language comprehension in real-time. For example, in visual search tasks, incremental spoken delivery of the target features (e.g., "Is there a red vertical?") can increase the efficiency of conjunction search because only one feature is heard at a time. Moreover, in spoken word recognition tasks, the visual presence of an object whose name is similar to the word being spoken (e.g., a candle present when instructed to "pick up the candy") can alter the process of comprehension. Dense sampling methods, such as eye-tracking and reach-tracking, richly illustrate the nature of this interaction, providing a semi-continuous measure of the temporal dynamics of individual behavioral responses. We review a variety of studies that demonstrate how these methods are particularly promising in further elucidating the dynamic competition that takes place between underlying linguistic and visual representations in multimodal contexts, and we conclude with a discussion of the consequences that these findings have for theories of embodied cognition.

Different mechanisms for role relations versus verb–action congruence effects: Evidence from ERPs in picture–sentence verification

“…In general terms, it would appear then that a model in which "utterance meaning, scene information, and linguistic expectation are representationally indistinguishable and reside within a unitary system that learns, represents, and processes language and the world" would fail to explain our results (Altmann & Mirković, 2009, p. 593). They are, by contrast, compatible with models that postulate a rapid interaction between linguistic and non-linguistic information (e.g., Anderson et al, 2011;Tanenhaus et al, 1995).…”

Referential and Visual Cues to Structural Choice in Visually Situated Sentence Production

“…In other words, what people currently have in mind can affect what they attend to later. Importantly, these memorial cues do not have to be spatial, as linguistic information currently held in working memory has also recently been shown to determine the spatial deployment of visual attention (e.g., Soto and Humphreys, 2007; Hodgson et al, 2009; Mannan et al, 2010; Anderson et al, 2011; Salverda and Altmann, 2011). Our data suggest that once the attentional cue is established in the speaker’s working memory, irrespective of whether it was established with the help of a pointer or a referent preview, this attentional cue biases the speaker to select the referent that later appears in the cued location as the sentential Subject.…”

An amodal shared resource model of language-mediated visual attention

“…Such multimodal interactions within the speaker and listener have been shown to be vital for language development (Markman, 1994; Bloom, 2000; Monaghan and Mattock, 2012; Mani et al, 2013) as well as for adult sentence and discourse processing (Anderson et al, 2011; Huettig et al, 2011b; Lupyan, 2012). Eye gaze has been used to demonstrate the nature of the processes supporting online integration of linguistic and visual information (Halberda, 2006; Huettig et al, 2011a).…”