Artificial neural network language models predict human brain responses to language even after a developmentally realistic amount of training

Abstract

Artificial neural networks have emerged as computationally plausible models of human language processing. A major criticism of these models is that the amount of training data they receive far exceeds that of humans during language learning. Here, we use two complementary approaches to ask how the models ability to capture human neural and behavioral responses to language is affected by the amount of training data. First, we evaluate GPT-2 models trained on 1 million, 10 million, 100 million, or 1 billion tokens against two fMRI benchmarks and one behavioral (reading times) benchmark. Because children are exposed to approximately 100 million words during the first 10 years of life, we consider the 100 million token model developmentally plausible. Second, we test the performance of a GPT-2 model that is trained on a 9 billion dataset to reach state-of-the-art next-word prediction performance against the same human benchmarks at different stages during training. Across both approaches, we find that (i) the models trained on a developmentally plausible amount of data already achieve near-maximal performance in capturing neural and behavioral responses to language. Further, (ii) lower perplexity (a measure of next-word prediction performance) is associated with stronger alignment with the human benchmarks, suggesting that models that have received enough training to achieve sufficiently high next word prediction performance also acquire human-like representations of the linguistic input. In tandem, these findings establish that although some training is necessary for the models ability to predict human responses to language, a developmentally realistic amount of training (~100 million tokens) may suffice.

Driving and suppressing the human language network using large language models

“…A number of studies have now shown that representations extracted from neural network models of language can capture neural responses to language in human brains, as recorded with fMRI or intracranial methods (e.g., [134][135][136][46][47][48][49][50][51][52]55,58 ). This study goes beyond prior work in four ways.…”

“…Second, we evaluated our encoding model on new individuals. Most studies that have used language models to model human neural responses to language (e.g., 135,136,46,[48][49][50][51][52]55,53,54,57,58 ; cf. 56,59 ) fit an encoding model using a given participant's data and then use that model to predict responses to held-out stimuli in that same participant.…”

“…Impressed by these models' successes, language researchers began using these large language models (LLMs) as models of human language processing. Strikingly, despite the fact that these models were not developed with the goal of modeling human language, some of these models (especially the unidirectional Transformer architectures; 36 ) appear to be able to capture human data, both behavioral (e.g., [42][43][44][45] ) and neural (e.g., [46][47][48][49][50][51][52][53][54][55][56][57][58] ). As such, this new class of computationally precise large language models (LLMs) has allowed the field of neurobiology of language to move beyond verbal descriptive models of language processing with promise of major advances in our understanding of human language processing.…”

Correspondence between the layered structure of deep language models and temporal structure of natural language processing in the human brain

“…An open question is how DLMs will perform when trained on more humanlike input: data that are not textual but spoken, multimodal, embodied and immersed in social actions. Interestingly, two recent papers suggest that language models trained on more realistic human-centered data can learn a language like children (53,54). However, additional research is needed to explore these questions.…”

Large Language Models Demonstrate the Potential of Statistical Learning in Language

“…In an even more recent preprint, Goldstein et al (2022b) found a correspondence between GPT-2's per-layer activation and the time course of language processing using EcoG (i.e., recordings from electrodes implanted in the brain). Importantly, this processing parallel does not seem to be a function of the unrealistic number of tokens to which LLMs are exposed during training: Hosseini et al (2022) found that GPT-2 embeddings can be used to predict the activation of the language network in an fMRI neuroimaging study even when trained on 100 million tokens or the equivalent of the first 10 years of a child's language exposure (Gilkerson et al, 2017). All this considering that the scale of GPT-2 is orders of magnitude smaller than the most recent wave of LLMs (Dettmers et al, 2022).…”