Speech-accompanying gestures are not processed by the language-processing mechanisms

Jouravlev, Olessia; Zheng, David X.; Balewski, Zuzanna; Pongos, Alvince L.; Levan, Zena; Goldin‐Meadow, Susan; Fedorenko, Evelina

doi:10.1016/j.neuropsychologia.2019.107132

Cited by 40 publications

(36 citation statements)

References 100 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our findings, along with prior findings from math and logic (Amalric & Dehaene, 2019;Monti et al, 2009Monti et al, , 2012, argue against this possibility: the language system does not respond to meaningful structured input that is non-linguistic. The lack of the language system engagement during code comprehension adds to the body of work that demonstrates high input selectivity of these regions (Fedorenko et al, 2011;Jouravlev et al, 2019;Monti et al, 2009Monti et al, , 2012Pritchett et al, 2018). Although the language system does not appear to support code comprehension, it may play a role in learning to program (Prat et al, 2020).…”

Section: The Language System Is Functionally Conservativementioning

confidence: 99%

“…Finally, comprehension of natural language recruits a set of left frontal and temporal brain regions known as the language system (e.g., Fedorenko & Thompson-Schill, 2014). These regions respond robustly to linguistic input (visual or auditory; Bemis & Pylkkänen, 2013;Deniz et al, 2019;Fedorenko et al, 2010) but show little or no response to tasks in non-linguistic domains, such as executive functions, math, logic, music, action observation, or non-linguistic communicative signals, such as gestures (Fedorenko et al, 2011;Jouravlev et al, 2019;Monti et al, 2009Monti et al, , 2012Pritchett et al, 2018;see Fedorenko & Blank, 2020, for a review). If code comprehension relies on the same circuits that map form to meaning in natural language, we expect to see activity within the language system.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Comprehension of computer code relies primarily on domain-general executive brain regions

Ivanova

Srikant

Sueoka

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Computer programming is a novel cognitive tool that has transformed modern society. An integral part of programming is code comprehension: the ability to process individual program tokens, combine them into statements, which, in turn, combine to form a program. What cognitive and neural mechanisms support this ability to process computer code? Here, we used fMRI to investigate the role of two candidate brain systems in code comprehension: the multiple demand (MD) system, typically recruited for math, logic, problem solving, and executive function, and the language system, typically recruited for linguistic processing. Across two experiments, we examined brain responses to code written in two programing languages: Python, a text-based programming language (Experiment 1) and ScratchJr, a graphical programming language for children (Experiment 2). To isolate neural activity evoked by code comprehension per se rather than by processing program content, we contrasted responses to code problems with responses to contentmatched sentence problems. We found that the MD system exhibited strong bilateral responses to code in both experiments. In contrast, the language system responded strongly to sentence problems, but only weakly or not at all to code problems. We conclude that code comprehension relies primarily on domaingeneral executive resources, demonstrating that the MD system supports the use of novel cognitive tools even when the input is structurally similar to natural language.

show abstract

Section: The Language System Is Functionally Conservativementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Comprehension of computer code relies primarily on domain-general executive brain regions

Ivanova

Srikant

Sueoka

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Each of the 30 experiments was originally designed to evaluate a specific hypothesis about (i) the sensitivity of the language and/or the MD network to some linguistic (lexical, syntactic, semantic, or pragmatic) manipulation, or (ii) the selectivity of the two networks for linguistic vs. non-linguistic conditions. For example, Experiment 2 compared responses to one-liner jokes vs. closely matched non-joke controls (Kline et al, submitted); Experiment 6 compared responses to syntactically simper vs. more complex sentences (sentences containing subject-vs. object-extracted relative clauses) ; Experiment 16 compared responses to spoken linguistic materials vs. speechaccompanying gestures (Jouravlev et al, 2019); and Experiment 24 contrasted sentences that contained a temporary syntactic ambiguity vs. control unambiguous sentences (following the design of Snijders et al, 2009). Data from some of these experiments have been published or are reported in preprints and papers under review (see Table 1); other experiments are parts of ongoing projects and have not yet been reported anywhere (we make the data used in the analyses below-estimates of neural responses to the conditions of the critical and localizer experiments in the MD and language fROIs in each participant (Table SI-2)-as well as whole-brain activation maps for the critical and localizer contrasts, available at https://osf.io/pdtk9/; raw data are available from the senior author upon request).…”

Section: Design Stimuli and Proceduresmentioning

confidence: 99%

“…Converging evidence from neuroimaging and patient studies suggests that a leftlateralized fronto-temporal brain network is selective for language processing. These regions respond to linguistic input (visual or auditory) across diverse materials and tasks (e.g., Fedorenko et al, 2010Vagharchakian et al, 2012;Scott et al, 2017;Deniz et al, 2019), but not to non-linguistic cognitive tasks, like arithmetic calculations, executive function tasks, music perception, action/gesture observation, and non-verbal social information (e.g., Fedorenko et al, 2011;Monti et al, 2012;Pritchett et al, 2018;Jouravlev et al, 2019;Paunov, 2019;see Fedorenko & Varley, 2016, for a review).…”

Section: Introduction (699 Words)mentioning

confidence: 99%

The domain-general multiple demand (MD) network does not support core aspects of language comprehension: a large-scale fMRI investigation

Diachek

Blank

Siegelman

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

AbstractAside from the language-selective left-lateralized fronto-temporal network, language comprehension sometimes additionally recruits a domain-general bilateral fronto-parietal network implicated in executive functions: the multiple demand (MD) network. However, the nature of the MD network’s contributions to language comprehension remains debated. To illuminate the role of this network in language processing, we conducted a large-scale fMRI investigation using data from 30 diverse word and sentence comprehension experiments (481 unique participants, 678 scanning sessions). In line with prior findings, the MD network was active during many language tasks. Moreover, similar to the language-selective network, which is robustly lateralized to the left hemisphere, these responses were stronger in the left-hemisphere MD regions. However, in stark contrast with the language-selective network, the MD network responded more strongly (i) to lists of unconnected words than to sentences, and critically, (ii) in paradigms with an explicit task compared to passive comprehension paradigms. In fact, many passive comprehension tasks failed to elicit a response above the fixation baseline in the MD network, in contrast to strong responses in the language-selective network. In tandem, these results argue against a role for the MD network in core aspects of sentence comprehension like inhibiting irrelevant meanings or parses, keeping intermediate representations active in working memory, or predicting upcoming words or structures. These results align with recent evidence of relatively poor tracking of the linguistic signal by the MD regions during naturalistic comprehension, and instead suggest that the MD network’s engagement during language processing likely reflects effort associated with extraneous task demands.Significance StatementDomain-general executive processes, like working memory and cognitive control, have long been implicated in language comprehension, including in neuroimaging studies that have reported activation in domain-general multiple demand (MD) regions for linguistic manipulations. However, much prior evidence has come from paradigms where language interpretation is accompanied by extraneous tasks. Using a large fMRI dataset (30 experiments/481 participants/678 sessions), we demonstrate that MD regions are engaged during language comprehension in the presence of task demands, but not during passive reading/listening—conditions that strongly activate the fronto-temporal language network. These results present a fundamental challenge to proposals whereby linguistic computations, like inhibiting irrelevant meanings, keeping representations active in working memory, or predicting upcoming elements, draw on domain-general executive resources.

show abstract

“…Recent evidence from neuroscience suggests that language processing is largely distinct from other aspects of cognition (Fedorenko & Blank, 2020;Fedorenko & Varley, 2016). A network of frontal and temporal brain regions (here referred to as the 'language network') has been found to respond to written/spoken/signed words and sentences, but not to mental arithmetic, music perception, executive function tasks, action/gesture perception, or computer programming (Amalric & Dehaene, 2019;Fedorenko et al, 2011;Ivanova et al, 2020;Jouravlev et al, 2019;Liu et al, 2020;Monti et al, 2009Monti et al, , 2012Pritchett et al, 2018). Similarly, investigations of patients with profound disruption of language capacity (global aphasia) have shown that these individuals can solve arithmetic and logic problems, appreciate music, and think about others' thoughts in spite of their language impairments (Basso & Capitani, 1985;Luria et al, 1965;Varley et al, 2005;Varley & Siegal, 2000), providing converging evidence that language and thought are neurally distinct.…”

Section: Introductionmentioning

confidence: 99%

The language network is recruited but not required for non-verbal event semantics

Ivanova

Mineroff

Zimmerer

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

Consistent with longstanding findings from neuropsychology, several brain regions in left frontal and temporal cortex respond robustly and selectively to language [1]. These regions, often referred to as the "language network", respond more strongly to meaningful stimuli (like words and sentences) than to stimuli devoid of meaning (like pseudowords and Jabberwocky sentences) [2]. But are these regions selectively recruited in processing linguistic meaning? Or do they instead store and/or process complex semantic information independent of its format (sentences or pictures)? In Experiment 1, we scanned participants with fMRI while they performed a semantic plausibility judgment task vs. a difficult perceptual control task on sentences and line drawings that describe/depict simple agent-patient interactions. We found that the language regions responded more strongly when participants performed the semantic task compared to the perceptual task, for both sentences and pictures (although sentences elicited overall stronger responses). Thus, healthy adults engage language regions when processing non-verbal meanings. But is this engagement necessary for understanding pictorial depictions of events? In Experiment 2, we tested two individuals with global aphasia, who have sustained massive damage to perisylvian language areas and display severe language difficulties, together with a group of control participants. Individuals with aphasia were at chance on a task of matching the sentences and pictures. However, they performed close to controls in assessing the plausibility of pictorial depictions of agent-patient interactions. Taken together, these results indicate that the left fronto-temporal language system is recruited but not necessary for processing complex non-verbal meanings.

show abstract

Speech-accompanying gestures are not processed by the language-processing mechanisms

Cited by 40 publications

References 100 publications

Comprehension of computer code relies primarily on domain-general executive brain regions

Comprehension of computer code relies primarily on domain-general executive brain regions

The domain-general multiple demand (MD) network does not support core aspects of language comprehension: a large-scale fMRI investigation

The language network is recruited but not required for non-verbal event semantics

Contact Info

Product

Resources

About