How Well Do Contrastive Learning Algorithms Model Human Real-time and Life-long Learning?

Zhuang, Chengxu; Xiang, Violet; Bai, Yoon; Jia, Xiaoxuan; DiCarlo, James J.; Yamins, Daniel

doi:10.32470/ccn.2022.1155-0

Cited by 20 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Existing work validates the impact of training tasks on model behavior and representations. Even when restricted to training on ImageNet images, the training objective and/or data augmentation can affect how well models match human similarity judgments of images (Muttenthaler, Dippel, Linhardt, Vandermeulen, & Kornblith, 2023), categorization patterns , performance on real-time and life-long learning benchmarks (Zhuang et al, 2022), and feature preferences , and also how well they predict primate physiology and human fMRI (Konkle & Alvarez, 2022) data. Still, it is possible to enrich DNN training tasks much further, even for object categorization (Sun, Shrivastava, Singh, & Gupta, 2017).…”

Section: Detailed and Holisticmentioning

confidence: 99%

Deep problems with neural network models of human vision

Bowers¹,

Dujmović²,

Montero³

et al. 2022

Behav Brain Sci

View full text Add to dashboard Cite

Deep neural networks (DNNs) have had extraordinary successes in classifying photographic images of objects and are often described as the best models of biological vision. This conclusion is largely based on three sets of findings: (1) DNNs are more accurate than any other model in classifying images taken from various datasets, (2) DNNs do the best job in predicting the pattern of human errors in classifying objects taken from various behavioral datasets, and (3) DNNs do the best job in predicting brain signals in response to images taken from various brain datasets (e.g., single cell responses or fMRI data). However, these behavioral and brain datasets do not test hypotheses regarding what features are contributing to good predictions and we show that the predictions may be mediated by DNNs that share little overlap with biological vision. More problematically, we show that DNNs account for almost no results from psychological research. This contradicts the common claim that DNNs are good, let alone the best, models of human object recognition. We argue that theorists interested in developing biologically plausible models of human vision need to direct their attention to explaining psychological findings. More generally, theorists need to build models that explain the results of experiments that manipulate independent variables designed to test hypotheses rather than compete on making the best predictions. We conclude by briefly summarizing various promising modelling approaches that focus on psychological data.

show abstract

Section: Detailed and Holisticmentioning

confidence: 99%

Deep problems with neural network models of human vision

Bowers¹,

Dujmović²,

Montero³

et al. 2022

Behav Brain Sci

View full text Add to dashboard Cite

show abstract

“…Indeed, this approach has been shown to improve vision models’ ability to capture primate neural responses ( Mehrer et al, 2021 ). It will also be important to investigate the role of the learning algorithms that the models use and their training objective, as both likely affect the representations that the models learn (e.g., see Zhuang et al, 2022 , for evidence from vision). Specifically, Zhuang et al (2022) showed that in an object categorization task, the negative sampling objective function, which maximizes the similarity between objects in the same category while minimizing the similarity between objects in different categories in the internal representation of the model, can alleviate model failure in capturing human visual behavior, which occurs under the standard objective function.…”

Section: Discussionmentioning

confidence: 99%

“…It will also be important to investigate the role of the learning algorithms that the models use and their training objective, as both likely affect the representations that the models learn (e.g., see Zhuang et al, 2022 , for evidence from vision). Specifically, Zhuang et al (2022) showed that in an object categorization task, the negative sampling objective function, which maximizes the similarity between objects in the same category while minimizing the similarity between objects in different categories in the internal representation of the model, can alleviate model failure in capturing human visual behavior, which occurs under the standard objective function. This failure is due to the presence of categories that are infrequent in the training data, and this finding can be relevant for language, which also contains infrequent elements (words and constructions) amid more common ones.…”

Section: Discussionmentioning

confidence: 99%

Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training

Hosseini,

Schrimpf,

Zhang

et al. 2024

Neurobiology of Language

View full text Add to dashboard Cite

Artificial neural networks have emerged as computationally plausible models of human language processing. A major criticism of these models is that the amount of training data they receive far exceeds that of humans during language learning. Here, we use two complementary approaches to ask how the models’ ability to capture human fMRI responses to sentences is affected by the amount of training data. First, we evaluate GPT-2 models trained on 1 million, 10 million, 100 million, or 1 billion words against an fMRI benchmark. We consider the 100-million-word model to be developmentally plausible in terms of the amount of training data given that this amount is similar to what children are estimated to be exposed to during the first 10 years of life. Second, we test the performance of a GPT-2 model trained on a 9-billion-token dataset to reach state-of-the-art next-word prediction performance on the human benchmark at different stages during training. Across both approaches, we find that (i) the models trained on a developmentally plausible amount of data already achieve near-maximal performance in capturing fMRI responses to sentences. Further, (ii) lower perplexity—a measure of next-word prediction performance—is associated with stronger alignment with human data, suggesting that models that have received enough training to achieve sufficiently high next-word prediction performance also acquire representations of sentences that are predictive of human fMRI responses. In tandem, these findings establish that although some training is necessary for the models’ predictive ability, a developmentally realistic amount of training (∼100 million words) may suffice.

show abstract

“…Indeed, this approach has been shown to improve vision models’ ability to capture primate neural responses (Mehrer et al 2021). Further, it will be important to investigate the role of the learning algorithms that the models use and their training objective , as both likely affect the representations that the models learn (e.g., see Zhuang et al, 2022 for evidence from vision).…”

Section: Discussionmentioning

confidence: 99%

Artificial neural network language models predict human brain responses to language even after a developmentally realistic amount of training

Hosseini

Schrimpf

Zhang

et al. 2022

Preprint

View full text Add to dashboard Cite

Artificial neural networks have emerged as computationally plausible models of human language processing. A major criticism of these models is that the amount of training data they receive far exceeds that of humans during language learning. Here, we use two complementary approaches to ask how the models ability to capture human neural and behavioral responses to language is affected by the amount of training data. First, we evaluate GPT-2 models trained on 1 million, 10 million, 100 million, or 1 billion tokens against two fMRI benchmarks and one behavioral (reading times) benchmark. Because children are exposed to approximately 100 million words during the first 10 years of life, we consider the 100 million token model developmentally plausible. Second, we test the performance of a GPT-2 model that is trained on a 9 billion dataset to reach state-of-the-art next-word prediction performance against the same human benchmarks at different stages during training. Across both approaches, we find that (i) the models trained on a developmentally plausible amount of data already achieve near-maximal performance in capturing neural and behavioral responses to language. Further, (ii) lower perplexity (a measure of next-word prediction performance) is associated with stronger alignment with the human benchmarks, suggesting that models that have received enough training to achieve sufficiently high next word prediction performance also acquire human-like representations of the linguistic input. In tandem, these findings establish that although some training is necessary for the models ability to predict human responses to language, a developmentally realistic amount of training (~100 million tokens) may suffice.

show abstract

How Well Do Contrastive Learning Algorithms Model Human Real-time and Life-long Learning?

Cited by 20 publications

References 0 publications

Deep problems with neural network models of human vision

Deep problems with neural network models of human vision

Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training

Artificial neural network language models predict human brain responses to language even after a developmentally realistic amount of training

Contact Info

Product

Resources

About