Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?

Abstract

Several popular Transformer based language models have been found to be successful for text-driven brain encoding. However, existing literature leverages only pretrained text Transformer models and has not explored the efficacy of task-specific learned Transformer representations. In this work, we explore transfer learning from representations learned for ten popular natural language processing tasks (two syntactic and eight semantic) for predicting brain responses from two diverse datasets: Pereira (subjects reading sentences from paragraphs) and Narratives (subjects listening to the spoken stories). Encoding models based on task features are used to predict activity in different regions across the whole brain. Features from coreference resolution, NER, and shallow syntax parsing explain greater variance for the reading activity. On the other hand, for the listening activity, tasks such as paraphrase generation, summarization, and natural language inference show better encoding performance. Experiments across all 10 task representations provide the following cognitive insights: (i) language left hemisphere has higher predictive brain activity versus language right hemisphere, (ii) posterior medial cortex, temporoparieto-occipital junction, dorsal frontal lobe have higher correlation versus early auditory and auditory association cortex, (iii) syntactic and semantic tasks display a good predictive performance across brain regions for reading and listening stimuli resp.

Lexical-Semantic Content, Not Syntactic Structure, Is the Main Contributor to ANN-Brain Similarity of fMRI Responses in the Language Network

“…A number of independent studies have recently shown that representations from state-of-the-art ANN models—especially unidirectional transformer models—align well with brain responses of humans processing linguistic input (e.g., Caucheteux & King, 2022 ; Gauthier & Levy, 2019 ; Goldstein et al, 2022 ; Jain & Huth, 2018 ; Kumar et al, 2022 ; Merlin & Toneva, 2022 ; Millet et al, 2022 ; Oota et al, 2022 ; Pasquiou et al, 2022 ; Pereira et al, 2018 ; Schrimpf et al, 2021 ; Toneva & Wehbe, 2019 ). However, what makes ANN representations align with human neural responses to language has been little explored (cf.…”

“…In particular, contemporary ANN language models achieve impressive performance on a variety of linguistic tasks (e.g., Brown et al, 2020 ; Chowdhery et al, 2022 ; Devlin et al, 2018 ; Liu et al, 2019 ; OpenAI, 2023 ; Rae et al, 2021 ). Furthermore, representations extracted from ANN language models—especially unidirectional attention transformer architectures like GPT2 ( Radford et al, 2019 ) can explain substantial variance in brain activity recorded from the human language network using regression-based evaluation metrics (e.g., Caucheteux & King, 2022 ; Gauthier & Levy, 2019 ; Goldstein et al, 2022 ; Hosseini et al, 2022 ; Jain & Huth, 2018 ; Kumar et al, 2022 ; Oota et al, 2022 ; Pasquiou et al, 2022 ; Schrimpf et al, 2021 ; Toneva & Wehbe, 2019 ). This correspondence has been suggested to derive, at least in part, from the convergence of the ANNs’ linguistic representations with those in the human brain ( Caucheteux & King, 2022 ; Goldstein et al, 2022 ; Hosseini et al, 2022 ; Schrimpf et al, 2021 ), despite the vast differences in their learning and architecture (e.g., Huebner & Willits, 2021 ; Warstadt & Bowman, 2022 ).…”

The Cortical Representation of Language Timescales is Shared between Reading and Listening

“…By specifically modeling representations of linguistic features, our results suggest that some of the differences observed in Regev et al (2013) could indeed be due to non-linguistic processes such as high-level control. A separate study suggesting that brain representations of language differ between modalities compared brain responses to different types of stimuli for reading and listening: the stimuli used for reading experiments consisted of isolated sentences, whereas the stimuli used for listening experiments consisted of full narratives (Oota et al, 2022). This discrepancy perhaps explains why in Oota et al (2022), language models trained on higher-level tasks (e.g., summarization, paraphrase detection) were better able to predict listening than reading data.…”

“…A separate study suggesting that brain representations of language differ between modalities compared brain responses to different types of stimuli for reading and listening: the stimuli used for reading experiments consisted of isolated sentences, whereas the stimuli used for listening experiments consisted of full narratives (Oota et al, 2022). This discrepancy perhaps explains why in Oota et al (2022), language models trained on higher-level tasks (e.g., summarization, paraphrase detection) were better able to predict listening than reading data. Our study used matched stimuli for reading and listening experiments, and the similarities we observed highlight the importance of using narrativelength, naturalistic stimuli to elicit brain representations of high-level linguistic features (Deniz et al, 2023).…”

Tuning in to Neural Encoding: Linking Human Brain and Artificial Supervised Representations of Language

“…We select tasks for this work based on two major principles that have also been adopted by previous work [21]. First, tasks that require a diverse set of cognitive-linguistic skills are chosen.…”