Mechanisms and Representations of Language-Mediated Visual Attention

Huettig, Falk; Mishra, Ramesh Kumar; Olivers, Christian N. L.

doi:10.3389/fpsyg.2011.00394

Cited by 33 publications

(23 citation statements)

References 106 publications

(151 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although it is generally agreed that knowledge about visual features is crucial for our understanding of concepts referring to physical objects (e.g., Huettig, Mishra, & Olivers, 2012), the long-standing and crucial question of the nature of the representations underlying this knowledge remains controversial. The flexible conceptual processing account described in the Introduction predicts that different kinds of features are recruited in situations that require different kinds of information.…”

Section: The Present Studymentioning

confidence: 99%

A task-dependent causal role for low-level visual processes in spoken word comprehension.

Ostarek

Huettig

2017

Journal of Experimental Psychology: Learning, Memory, and Cogni

Self Cite

View full text Add to dashboard Cite

It is well established that the comprehension of spoken words referring to object concepts relies on high-level visual areas in the ventral stream that build increasingly abstract representations. It is much less clear whether basic low-level visual representations are also involved. Here we asked in what task situations low-level visual representations contribute functionally to concrete word comprehension using an interference paradigm. We interfered with basic visual processing while participants performed a concreteness task (Experiment 1), a lexical-decision task (Experiment 2), and a word class judgment task (Experiment 3). We found that visual noise interfered more with concrete versus abstract word processing, but only when the task required visual information to be accessed. This suggests that basic visual processes can be causally involved in language comprehension, but that their recruitment is not automatic and rather depends on the type of information that is required in a given task situation.

show abstract

Section: The Present Studymentioning

confidence: 99%

A task-dependent causal role for low-level visual processes in spoken word comprehension.

Ostarek

Huettig

2017

Journal of Experimental Psychology: Learning, Memory, and Cogni

Self Cite

View full text Add to dashboard Cite

show abstract

“…These studies provide insight into the type of information activated as a spoken word unfolds, the relative influence of specific sources of information during speech comprehension, and the temporal structure of this process. Such insights are based on the assumption that gaze towards an item reflects the level to which properties of the item (relative to all other items within the display) are activated at a given point in time by the speech signal (see Ferreira & Tanenhaus, 2007;Huettig, Mishra, & Olivers, 2012;Tanenhaus, Magnuson, Dahan, & Chambers, 2000).…”

Section: Models Of Multimodal Integration During Speech Processingmentioning

confidence: 99%

The multimodal nature of spoken word processing in the visual world: Testing the predictions of alternative models of multimodal integration

Smith

Monaghan

Huettig

2017

Journal of Memory and Language

Self Cite

View full text Add to dashboard Cite

a b s t r a c tAmbiguity in natural language is ubiquitous, yet spoken communication is effective due to integration of information carried in the speech signal with information available in the surrounding multimodal landscape. Language mediated visual attention requires visual and linguistic information integration and has thus been used to examine properties of the architecture supporting multimodal processing during spoken language comprehension. In this paper we test predictions generated by alternative models of this multimodal system. A model (TRACE) in which multimodal information is combined at the point of the lexical representations of words generated predictions of a stronger effect of phonological rhyme relative to semantic and visual information on gaze behaviour, whereas a model in which sub-lexical information can interact across modalities (MIM) predicted a greater influence of visual and semantic information, compared to phonological rhyme. Two visual world experiments designed to test these predictions offer support for sub-lexical multimodal interaction during online language processing.

show abstract

“…However, the nature of the information and mechanisms involved in visual world and language processing interactions are as yet underspecified (Huettig et al, 2011a, 2012). How is information activated within one modality integrated with information activated within another, what form does this information take, how does such information interact and how is this interaction connected to eye gaze behavior?…”

Section: Integrative Processing In a Model Of Language-mediated Visuamentioning

confidence: 99%

An amodal shared resource model of language-mediated visual attention

Smith¹,

Monaghan²,

Huettig³

2013

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

Language-mediated visual attention describes the interaction of two fundamental components of the human cognitive system, language and vision. Within this paper we present an amodal shared resource model of language-mediated visual attention that offers a description of the information and processes involved in this complex multimodal behavior and a potential explanation for how this ability is acquired. We demonstrate that the model is not only sufficient to account for the experimental effects of Visual World Paradigm studies but also that these effects are emergent properties of the architecture of the model itself, rather than requiring separate information processing channels or modular processing systems. The model provides an explicit description of the connection between the modality-specific input from language and vision and the distribution of eye gaze in language-mediated visual attention. The paper concludes by discussing future applications for the model, specifically its potential for investigating the factors driving observed individual differences in language-mediated eye gaze.

show abstract

Mechanisms and Representations of Language-Mediated Visual Attention

Cited by 33 publications

References 106 publications

A task-dependent causal role for low-level visual processes in spoken word comprehension.

A task-dependent causal role for low-level visual processes in spoken word comprehension.

The multimodal nature of spoken word processing in the visual world: Testing the predictions of alternative models of multimodal integration

An amodal shared resource model of language-mediated visual attention

Contact Info

Product

Resources

About