Fixing the stimulus-as-fixed-effect fallacy in task fMRI

Westfall, Jacob; Nichols, Thomas E.; Yarkoni, Tal

doi:10.12688/wellcomeopenres.10298.1

Cited by 53 publications

(71 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This creates a tension: experimentalists use contrived stimuli and designs to recover elegant coding principles (e.g., Hubel & Wiesel, 1962); but it remains unclear whether these principles actually capture neural responses in naturalistic contexts (Felsen & Dan, 2005;Olshausen & Field, 2005;Hasson & Honey, 2012;Hamilton & Huth, 2018). From a statistical perspective, this tension is also partially reflected in anxiety about the "stimulus-as-fixed-effect" fallacy in psycholinguistics (Coleman, 1964;Clark, 1973;Baayen et al, 2008), social psychology (Brunswik, 1955;Wells & Windschitl, 1999;Judd et al, 2012), and cognitive neuroscience (Bedny et al, 2007;Westfall et al, 2016). Historically these practices and tensions can in part be traced to an argument from cognitive psychology that the brain is not exposed to rich enough data from the environment to navigate the problem space (Chomsky, 1965).…”

Section: Generalization Based On Impoverished Datamentioning

confidence: 99%

Direct-fit to nature: an evolutionary perspective on biological (and artificial) neural networks

Hasson

Nastase

Goldstein

2019

Preprint

View full text Add to dashboard Cite

Evolution is a blind fitting process by which organisms, over generations, adapt to the niches of an ever-changing environment. Does the mammalian brain use similar brute-force fitting processes to learn how to perceive and act upon the world? Recent advances in training deep neural networks has exposed the power of optimizing millions of synaptic weights to map millions of observations along ecologically relevant objective functions. This class of models has dramatically outstripped simpler, more intuitive models, operating robustly in real-life contexts spanning perception, language, and action coordination. These models do not learn an explicit, human-interpretable representation of the underlying structure of the data; rather, they use local computations to interpolate over task-relevant manifolds in a high-dimensional parameter space. Furthermore, counterintuitively, over-parameterized models, similarly to evolutionary processes, can be simple and parsimonious as they provide a versatile, robust solution for learning a diverse set of functions. In contrast to traditional scientific models, where the ultimate goal is interpretability, over-parameterized models eschew interpretability in favor of solving real-life problems or tasks. We contend that over-parameterized blind fitting presents a radical challenge to many of the underlying assumptions and practices in computational neuroscience and cognitive psychology. At the same time, this shift in perspective informs longstanding debates and establishes unexpected links with evolution, ecological psychology, and artificial life. Simple versus multidimensional modelsAs with any scientific model, neuroscientific models are often judged based on their interpretability (i.e., providing an explicit, formulaic description of the underlying causes) and generalization (i.e., the capacity for prediction over broad, novel contexts; e.g., von Neumann, 1955 ). However, in practice, interpretability and 2 generalization are often at odds: interpretable models may have considerable explanatory appeal but poor predictive power, while high-performing predictive models may be difficult to interpret (Breiman, 2001;Shmueli, 2010;Yarkoni and Westfall, 2017). This tension is particularly striking when modeling brain and behavior. The brain itself, in orchestrating behavior, is by conventional standards a wildly over-

show abstract

Section: Generalization Based On Impoverished Datamentioning

confidence: 99%

Direct-fit to nature: an evolutionary perspective on biological (and artificial) neural networks

Hasson

Nastase

Goldstein

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Note that such an item-wise approach differs from the typical method of assessing such 2 nd -order correlations between brain and model RDMs (Kriegeskorte and Kievit 2013;Clarke and Tyler 2014), which typically relate the entire item x item matrix at once, and thus generalize across all items that comprise the matrix, and furthermore do not explicitly assess the model fit or error associated with such a brain-behavior comparison. This more general approach therefore handicaps any attempt to capture both the predictive value of item-specific 2 nd -order similarity, as well as any attempt to capture the variation of model stimuli as a random effect (Westfall et al 2016), which we model explicitly below within the context of a mixed-effects logistic model across all subjects and items. Thus, the IRAFs for each visual and semantic RDM were used as predictors in a mixed-effects logistic regression analysis to predict subsequent memory [0,1] for items that were remembered in the conceptual but not the perceptual memory test (Conceptual Memory) vs. items that were remembered in the perceptual but not the conceptual memory test (Perceptual Memory).…”

Section: Identifying Brain Regions Where the Rdm-activity Fit (Iraf) mentioning

confidence: 99%

Visual and semantic representations predict subsequent memory in perceptual and conceptual memory tests

Davis

Geib

Wing

et al. 2020

Preprint

View full text Add to dashboard Cite

Acknowledgments:The authors would like to thank Aude Oliva and Wilma Bainbridge for help in the conception of this project, and Alex Clarke for helpful comments on the manuscript. AbstractIt is generally assumed that the encoding of a single event generates multiple memory representations, which contribute differently to subsequent episodic memory. We used fMRI and representational similarity analysis (RSA) to examine how visual and semantic representations predicted subsequent memory for single item encoding (e.g., seeing an orange). Three levels of visual representations corresponding to early, middle, and late visual processing stages were based on a deep neural network. Three levels of semantic representations were based on normative Observed ("is round"), Taxonomic ("is a fruit"), and Encyclopedic features ("is sweet"). We identified brain regions where each representation type predicted later Perceptual Memory, Conceptual Memory, or both (General Memory). Participants encoded objects during fMRI, and then completed both a word-based conceptual and picture-based perceptual memory test. Visual representations predicted subsequent Perceptual Memory in visual cortices, but also facilitated Conceptual and General Memory in more anterior regions. Semantic representations, in turn, predicted Perceptual Memory in visual cortex, Conceptual Memory in the perirhinal and inferior prefrontal cortex, and General Memory in the angular gyrus. These results suggest that the contribution of visual and semantic representations to subsequent memory effects depends on a complex interaction between representation, test type, and storage location.

show abstract

“…Each regressor consisted of a single stick function (also known as a delta function or tent function) at the appropriate post-stimulus time, and was zero everywhere else, so that fitting the entire set of regressors modeled the time course of the response without any assumptions about its shape. As shown by the design matrix in Figure 1 In event-related fMRI, a single stimulus condition usually consists of presentation of multiple exemplars (for instance, a face, followed a few seconds later by a different face image, and so on); the deconvolved impulse response function represent the average response across all faces even though it is understood that individual stimuli evoke differing responses (Westfall et al, 2016). Our implementation of iEEG deconvolution is equivalent to this approach: the deconvolved visual, auditory, and interaction responses represent the average response across the four different word stimuli.…”

Section: Ieeg Data Analysis: Deconvolutionmentioning

confidence: 99%

Responses to Visual Speech in Human Posterior Superior Temporal Gyrus Examined with iEEG Deconvolution

Metzger

Magnotti

Wang

et al. 2020

Preprint

View full text Add to dashboard Cite

Acknowledgments:This research was supported by NIH R01NS065395 and R25NS070694. AbstractExperimentalists studying multisensory integration compare neural responses to multisensory stimuli with responses to the component modalities presented in isolation. This procedure is problematic for multisensory speech perception since audiovisual speech and auditory-only speech are easily intelligible but visual-only speech is not. To overcome this confound, we developed intracranial encephalography (iEEG) deconvolution. Individual stimuli always contained both auditory and visual speech but jittering the onset asynchrony between modalities allowed for the time course of the unisensory responses and the interaction between them to be independently estimated. We applied this procedure to electrodes implanted in human epilepsy patients (both male and female) over the posterior superior temporal gyrus (pSTG), a brain area known to be important for speech perception. iEEG deconvolution revealed sustained, positive responses to visual-only speech and larger, phasic responses to auditory-only speech. Confirming results from scalp EEG, responses to audiovisual speech were weaker than responses to auditoryonly speech, demonstrating a subadditive multisensory neural computation. Leveraging the spatial resolution of iEEG, we extended these results to show that subadditivity is most pronounced in more posterior aspects of the pSTG. Across electrodes, subadditivity correlated with visual responsiveness, supporting a model in visual speech enhances the efficiency of auditory speech processing in pSTG. The ability to separate neural processes may make iEEG deconvolution useful for studying a variety of complex cognitive and perceptual tasks. Significance statementUnderstanding speech is one of the most important human abilities. Speech perception uses information from both the auditory and visual modalities. It has been difficult to study neural responses to visual speech because visual-only speech is difficult or impossible to comprehend, unlike auditory-only and audiovisual speech. We used intracranial encephalography (iEEG) deconvolution to overcome this obstacle. We found that visual speech evokes a positive response in the human posterior superior temporal gyrus, enhancing the efficiency of auditory speech processing.

show abstract

Fixing the stimulus-as-fixed-effect fallacy in task fMRI

Cited by 53 publications

References 30 publications

Direct-fit to nature: an evolutionary perspective on biological (and artificial) neural networks

Direct-fit to nature: an evolutionary perspective on biological (and artificial) neural networks

Visual and semantic representations predict subsequent memory in perceptual and conceptual memory tests

Responses to Visual Speech in Human Posterior Superior Temporal Gyrus Examined with iEEG Deconvolution

Contact Info

Product

Resources

About