A Visuospatial Dataset for Naturalistic Verb Learning

Abstract

We introduce a new dataset for training and evaluating grounded language models. Our data is collected within a virtual reality environment and is designed to emulate the quality of language data to which a pre-verbal child is likely to have access: That is, naturalistic, spontaneous speech paired with richly grounded visuospatial context. We use the collected data to compare several distributional semantics models for verb learning. We evaluate neural models based on 2D (pixel) features as well as feature-engineered models based on 3D (symbolic, spatial) features, and show that neither modeling approach achieves satisfactory performance. Our results are consistent with evidence from child language acquisition that emphasizes the difficulty of learning verbs from naive distributional data. We discuss avenues for future work on cognitively-inspired grounded language learning, and release our corpus with the intent of facilitating research on the topic.1 https://github.com/dylanebert/nbc 2 Our university namesake, plus paying homage to important Brown corpora in both NLP (Francis and Kucera, 1979) and Child Language Acquisition (Brown, 1973).

Cross‐Situational Word Learning With Multimodal Neural Networks

“…Finally, while we attempted to test a broad range of phenomena, our list was by no means exhaustive. Future work should aim to examine other aspects of word learning not considered here, such as visual grounding of referents from other kinds of lexical classes such as verbs and adjectives (Ebert & Pavlick, 2020;Nikolaus & Fourtassi, 2021). 6 Although it is common in cross-situational word learning experiments to match the number of words and referents per trial, this form of similarity allows some additional flexibility in handling situations where the number of words differs from the number of referents, like when the set of words is a sentence in natural language and not every word can be mapped onto a visually grounded referent.…”