How direct is the link between words and images?

Abstract

Günther et al. (2022) investigated the relationship between words and images in which they concluded the possibility of a direct link between words and embodied experience. In their study, participants were presented with a target noun and a pair of images, one chosen by their model and another chosen randomly. Participants were asked to select the image that best matched the target noun. Building upon their work, we addressed the following questions. 1. Apart from utilizing visually embodied simulation, what other strategies subjects might have used? How much does this setup rely on visual information? Can it be solved using textual representations? 2. Do current visually-grounded embeddings explain subjects’ selection behavior better than textual embeddings? 3. Does visual grounding improve the representations of both concrete and abstract words? For this aim, we designed novel experiments based on pre-trained word embeddings. Our experiments reveal that subjects’ selection behavior is explained to a large extend on text-based embeddings and word-based similarities. Visually grounded embeddings offered modest advantages over textual embeddings in certain cases. These findings indicate that the experiment by Günther et al. (2022) may not be well suited for tapping into the perceptual experience of participants, and the extent to which it measures visually grounded knowledge is unclear.

One Cue's Loss Is Another Cue's Gain—Learning Morphophonology Through Unlearning

“…These studies use one-hot encoded vectors to represent phonology, and word embeddings to represent meaning, replacing the Rescorla-Wagner equations with the similar, but computationally more powerful Widrow-Hoff delta rule (Widrow & Hoff, 1960). In the context of our study, MEL-frequency bands (Shafaei-Bajestan et al, 2023) could be used to represent the auditory stimuli and visual embeddings (Shahmohammadi, Heitmeier, Shafaei-Bajestan, Lensch, & Baayen, 2024) to represent the visual stimuli. We expect that the unlearning effect we have observed will also emerge with such continuous representations.…”