Learning Through Processing: Toward an Integrated Approach to Early Word Learning

Meylan, Stephan C.; Bergelson, Elika

doi:10.1146/annurev-linguistics-031220-011146

Cited by 11 publications

(12 citation statements)

References 128 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although our primary aim was establishing the learnability of word-referent mappings with minimal ingredients, CVCL’s successes do not rule out more sophisticated forms of representation and reasoning, especially ones that might emerge in later development ( 55 ). These include mutual exclusivity ( 13 ), the principle of contrast ( 12 ), the shape bias ( 56 ), syntactic cues ( 57 ), social or gestural cues ( 15 ), or hypothesis generation ( 58 ).…”

Section: Discussionmentioning

confidence: 99%

Grounded language acquisition through the eyes and ears of a single child

Vong,

Wang,

Orhan

et al. 2024

Science

View full text Add to dashboard Cite

Starting around 6 to 9 months of age, children begin acquiring their first words, linking spoken words to their visual counterparts. How much of this knowledge is learnable from sensory input with relatively generic learning mechanisms, and how much requires stronger inductive biases? Using longitudinal head-mounted camera recordings from one child aged 6 to 25 months, we trained a relatively generic neural network on 61 hours of correlated visual-linguistic data streams, learning feature-based representations and cross-modal associations. Our model acquires many word-referent mappings present in the child’s everyday experience, enables zero-shot generalization to new visual referents, and aligns its visual and linguistic conceptual systems. These results show how critical aspects of grounded word meaning are learnable through joint representation and associative learning from one child’s input.

show abstract

Section: Discussionmentioning

confidence: 99%

Grounded language acquisition through the eyes and ears of a single child

Vong,

Wang,

Orhan

et al. 2024

Science

View full text Add to dashboard Cite

show abstract

“…Our multimodal training setup also does not capture the full richness of the multimodal signals that children may receive. Beyond imperfections in preprocessing (Section 2) and the inherent stochasticity in a child's gaze (Yu, Zhang, Slone, & Smith, 2021), the use of tokenized text rather than audio removes phonological or morphological cues, while also treating segmentation capabilities as given (Meylan & Bergelson, 2022). We mainly focused on linguistic analyses that are applicable to text‐only setups, because this enables us to better isolate the contribution of introducing multimodality.…”

Section: Discussionmentioning

confidence: 99%

Finding Structure in One Child's Linguistic Experience

et al. 2023

View full text Add to dashboard Cite

Neural network models have recently made striking progress in natural language processing, but they are typically trained on orders of magnitude more language input than children receive. What can these neural networks, which are primarily distributional learners, learn from a naturalistic subset of a single child's experience? We examine this question using a recent longitudinal dataset collected from a single child, consisting of egocentric visual data paired with text transcripts. We train both languageonly and vision-and-language neural networks and analyze the linguistic knowledge they acquire. In parallel with findings from Jeffrey Elman's seminal work, the neural networks form emergent clusters of words corresponding to syntactic (nouns, transitive and intransitive verbs) and semantic categories (e.g., animals and clothing), based solely on one child's linguistic input. The networks also acquire sensitivity to acceptability contrasts from linguistic phenomena, such as determiner-noun agreement and argument structure. We find that incorporating visual information produces an incremental gain in predicting words in context, especially for syntactic categories that are comparatively more easily grounded, such as nouns and verbs, but the underlying linguistic representations are not fundamentally altered. Our findings demonstrate which kinds of linguistic knowledge are learnable from a snapshot of a single child's real developmental experience.

show abstract

“…Other work has explored this problem of scalability in a variety of ways, from early multimodal approaches (Roy & Pentland, 2002), to more recent work using large-scale naturalistic headcam data (Orhan, Gupta, & Lake, 2020;Tsutsui, Chandrasekaran, Reza, Crandall, & Yu, 2020) and studying the ways in which children or machines an active role in word learning (Gelderloos, Kamelabad, & Alishahi, 2020;Zettersten & Saffran, 2019). The fact multimodal neural networks can be trained from scratch, as demonstrated in Experiment 7 and other works (Harwath et al, 2018;Radford et al, 2021), suggests that these kinds of networks could be further developed to provide a unifying account of artificial word learning in the lab and naturalistic word learning in the wild (Meylan & Bergelson, 2021). Finally, while we attempted to test a broad range of phenomena, our list was by no means exhaustive.…”

Section: Discussionmentioning

confidence: 99%

Cross‐Situational Word Learning With Multimodal Neural Networks

Vong

Lake

2022

Cognitive Science

View full text Add to dashboard Cite

In order to learn the mappings from words to referents, children must integrate co-occurrence information across individually ambiguous pairs of scenes and utterances, a challenge known as crosssituational word learning. In machine learning, recent multimodal neural networks have been shown to learn meaningful visual-linguistic mappings from cross-situational data, as needed to solve problems such as image captioning and visual question answering. These networks are potentially appealing as cognitive models because they can learn from raw visual and linguistic stimuli, something previous cognitive models have not addressed. In this paper, we examine whether recent machine learning approaches can help explain various behavioral phenomena from the psychological literature on crosssituational word learning. We consider two variants of a multimodal neural network architecture and look at seven different phenomena associated with cross-situational word learning and word learning more generally. Our results show that these networks can learn word-referent mappings from a single epoch of training, mimicking the amount of training commonly found in cross-situational word learning experiments. Additionally, these networks capture some, but not all of the phenomena we studied, with all of the failures related to reasoning via mutual exclusivity. These results provide insight into the kinds of phenomena that arise naturally from relatively generic neural network learning algorithms, and which word learning phenomena require additional inductive biases.

show abstract

Learning Through Processing: Toward an Integrated Approach to Early Word Learning

Cited by 11 publications

References 128 publications

Grounded language acquisition through the eyes and ears of a single child

Grounded language acquisition through the eyes and ears of a single child

Finding Structure in One Child's Linguistic Experience

Cross‐Situational Word Learning With Multimodal Neural Networks

Contact Info

Product

Resources

About