Where word and world meet: Language and vision share an abstract representation of symmetry.

Abstract

Symmetry is ubiquitous in nature, in logic and mathematics, and in perception, language, and thought. Although humans are exquisitely sensitive to visual symmetry (e.g., of a butterfly), symmetry in natural language goes beyond visuospatial properties: many words point to abstract concepts with symmetrical content (e.g., equal, marry). For example, if Mark marries Bill, then Bill marries Mark. In both cases (vision and language), symmetry may be formally characterized as invariance under transformation. Is this a coincidence, or is there some deeper psychological resemblance? Here we asked whether representations of symmetry correspond across language and vision. To do so, we developed a novel cross-modal matching paradigm. On each trial, participants observed a visual stimulus (either symmetrical or nonsymmetrical) and had to choose between a symmetrical and nonsymmetrical English predicate unrelated to the stimulus (e.g., "negotiate" vs. "propose"). In a first study with visual events (symmetrical collision or asymmetrical launch), participants reliably chose the predicate matching the event's symmetry. A second study showed that this "language-vision correspondence" generalized to objects and was weakened when the stimuli's binary nature was made less apparent (i.e., for one object, rather than two inward-facing objects). A final study showed the same effect when nonsigners guessed English translations of signs from American Sign Language, which expresses many symmetrical concepts spatially. Taken together, our findings support the existence of an abstract representation of symmetry which humans access via both perceptual and linguistic means. More broadly, this work sheds light on the rich, structured nature of the language-cognition interface.

Compositionality in visual perception

“…For example, it may partially explain how information from perception is 'readily consumed' by cognitive and linguistic systems (due to the similar formats of some perceptual and higher-level representations; Quilty-Dunn, 2020;Cavanagh, 2021). Recent work explores these connections explicitly: skeletal shape representations impact aesthetic preferences and linguistic descriptions of shapes (Sun & Firestone, 2022a, 2022b, and representations of symmetry and roles may be shared across perception and language (Strickland, 2017;Hafri et al, 2018;Rissman & Majid, 2019;Hafri et al, 2022). One could also investigate the 'psychophysics' of compositional processes-the timing/ordering of how relational representations are built from their parts.…”

Stop me if you've heard this one before: The Chomskyan hammer and the Skinnerian nail

“…Moreover, visual brain areas encode skeletal structure across surface changes (Ayzenberg, Kamps, Dilks, & Lourenco, 2022;Hung, Carlson, & Connor, 2012;Lescroart & Biederman, 2013). Skeletal representations may also encode nonmetric, categorical propertiesfor example, straight/curved and symmetric/asymmetric (Amir, Biederman, & Hayworth, 2012;Green, 2017;Hafri, Gleitman, Landau, & Trueswell, 2023).…”

“…For example, it may partially explain how information from perception is "readily consumed" by cognitive and linguistic systems (because of the similar formats of some perceptual and higher-level representations; Quilty-Dunn, 2020). Recent work explores these connections explicitly: Skeletal shape representations impact aesthetic preferences and linguistic descriptions of shapes (Sun & Firestone, 2022a, and representations of symmetry and roles may be shared across perception and language (Hafri et al, 2018(Hafri et al, , 2023Strickland, 2017). One could also investigate the "psychophysics" of compositional processes the timing and ordering of how relational representations are built from their parts.…”

Compositionality in visual perception

“…Moreover, visual brain areas encode skeletal structure across surface changes (Ayzenberg, Kamps, Dilks, & Lourenco, 2022; Hung, Carlson, & Connor, 2012; Lescroart & Biederman, 2013). Skeletal representations may also encode nonmetric, categorical properties – for example, straight / curved and symmetric / asymmetric (Amir, Biederman, & Hayworth, 2012; Green, 2017; Hafri, Gleitman, Landau, & Trueswell, 2023).…”

“…For example, it may partially explain how information from perception is “readily consumed” by cognitive and linguistic systems (because of the similar formats of some perceptual and higher-level representations; Cavanagh, 2021; Quilty-Dunn, 2020). Recent work explores these connections explicitly: Skeletal shape representations impact aesthetic preferences and linguistic descriptions of shapes (Sun & Firestone, 2022a, 2022b), and representations of symmetry and roles may be shared across perception and language (Hafri et al, 2018, 2023; Rissman & Majid, 2019; Strickland, 2017). One could also investigate the “psychophysics” of compositional processes – the timing and ordering of how relational representations are built from their parts.…”