A Cross‐Modal and Cross‐lingual Study of Iconicity in Language: Insights From Deep Learning

Abstract

The present paper addresses the study of non‐arbitrariness in language within a deep learning framework. We present a set of experiments aimed at assessing the pervasiveness of different forms of non‐arbitrary phonological patterns across a set of typologically distant languages. Different sequence‐processing neural networks are trained in a set of languages to associate the phonetic vectorization of a set of words to their sensory (Experiment 1), semantic (Experiment 2), and word‐class representations (Experiment 3). The models are then tested, without further training, in a set of novel instances in a language belonging to a different language family, and their performance is compared with a randomized baseline. We show that the three cross‐domain mappings can be successfully transferred across languages and language families, suggesting that the phonological structure of the lexicon is pervaded with language‐invariant cues about the words' meaning and their syntactic classes.

Prolegomena to a cognitive science of language diversity

“…They are conventionalized, structurally (usually, phonetically) marked words associated with vivid sensory images, which are usually employed to depict qualities of things or events and which are a grammatical category in only a small set of languages (see Dingemanse, 2012 for a review). The similarities found by typological research among the ideophones of the world languages is seemingly explained by their closer relationship to the sensory and motor systems within human cognition, which are rather invariant (De Varda & Strapparava, 2022). Yet, some withinand across-language variability in how ideophones are built and used has been documented too (Dingemanse, 2012).…”

Cracking Arbitrariness: A Data-driven Study of Auditory Iconicity

“…The employment of data-driven models in the study of perceptually-grounded meanings has gained increasing popularity in the last years, with a prominent example being set by the adoption of computer-vision deep-learning architectures in experimental psychology (Günther et al, 2020(Günther et al, , 2021Petilli et al, 2021) and neuroscience (Seeliger et al, 2018;Yamins & DiCarlo, 2016). These models have also been employed in iconicity research: de Varda & Strapparava (2022) have employed an image-processing neural network to show that word sounds bear a cross-modal resemblance to the visual representations of their referents, with this resemblance being consistent across languages. However, their study was focused on phonovisual iconicity, and thus it implied a cross-modal language-to-perception mapping.…”

“…This methodology, while overcoming the limitations of iconicity ratings, presents a different shortcoming. Similarly to other data-driven approaches to phonosymbolism (Blasi et al, 2016, see for instance), de Varda & Strapparava's (2022) study embraces a functional definition of iconicity, characterized as a feature of a signal that allows its meaning to be predicted from its form (Motamedi et al, 2019). Functional approaches which do not employ direct resemblance as a criterion can be problematic, as they may conflate iconicity and systematicity, a related but distinc phenomenon (Dingemanse et al, 2015).…”

THINGSplus: New norms and metadata for the THINGS database of 1854 object concepts and 26,107 natural object images

“…It supports researchers in selecting a representative and standardized set of object concepts and images, providing a foundation for exploring the perceptual and cognitive processing of complex real-world stimuli at scale or with a systematic sampling strategy. In addition, with THINGS starting to be adopted more widely (e.g., de Varda & Strapparava, 2022 ; Demircan et al, 2022 ; Dobs et al, 2022 ; Frey et al, 2021 ; Gifford et al, 2022 ; Griffin, 2019 ; Grootswagers et al, 2022 ; Lam et al, 2021 ; Muttenthaler et al, 2022 ; Ratan Murty et al, 2021 ; Rideaux et al, 2022 ), THINGS allows increased comparability between studies across different laboratories or disciplines.…”