Compositionality in visual perception

Abstract

Quilty-Dunn et al.'s wide-ranging defense of the Language of Thought Hypothesis (LoTH) argues that vision traffics in abstract, structured representational formats. We agree: Vision, like language, is compositional – just as words compose into phrases, many visual representations contain discrete constituents that combine in systematic ways. Here, we amass evidence extending this proposal, and explore its implications for how vision interfaces with the rest of the mind.

A Phone in a Basket Looks Like a Knife in a Cup: Role-Filler Independence in Visual Processing

“…The present results complement these developmental and cross-linguistic findings by showing that core subtypes of in and on are not only represented in cognition (in both infants and adults), but are also automatically extracted in visual processing. More generally, our results add to the growing evidence for abstract, categorical information shared by processes in development, language, and perception (Cavanagh, 2021 ; Hafri et al, 2023 ; Quilty-Dunn, 2020 ; Strickland, 2017 ).…”

“…In this way, visual processing may have an important and powerful property in common with certain forms of linguistic processing, namely, its compositional nature—containing discrete constituents that are combined in systematic (and often novel) ways. Just as the compositionality of language supports your ability to understand sentences you have never heard before (Chomsky, 1957 ; Jackendoff, 1987 ), and the abstractness of thought supports your ability to generate new thoughts (Fodor & Pylyshyn, 1988 ), the abstractness of relational perception may permit you to perceive instances of relations you have never seen before (for further discussion, see Hafri et al, 2023 ).…”

“…Crucially, however, none of this work has asked whether visual processing respects the core relational property of role-filler independence, i.e., whether it maintains the identity of certain relations apart from their participating entities. A positive answer to this question would shed new light not only on mechanisms of relational representation itself, but also on the kinds of contents perception can represent in the first place, and the formats used to represent them—perhaps not only iconic, analog, or “picture-like” (Block, 2023 ; Carey, 2009 ; Kosslyn et al, 2006 ) but also discrete, symbolic, or “sentence-like” (Hafri et al, 2023 ; Mandelbaum et al, 2022 ; Quilty-Dunn, 2020 ; Quilty-Dunn et al, 2022 )—a point to which we return in the General Discussion .…”