Making thebrain-activity-to-informationleap using a novel framework: Stimulus Information Representation (SIR)

Abstract

It is no good poking around in the brain without some idea of what one is looking for. That would be like trying to find a needle in a haystack without having any idea what needles look like. The theorist is the [person] who might reasonably be asked for [their] opinion about the appearance of needles." HC Longuet-Higgins, 1969. AbstractA fundamental challenge in neuroscience is to understand how the brain processes information. Neuroscientists have approached this question partly by measuring brain activity in space, time and at different levels of granularity. However, our aim is not to discover brain activity per se, but to understand the processing of information that this activity reflects. To make this brain-activityto-information leap, we believe that we should reconsider brain imaging from the methodological foundations of psychology. With this goal in mind, we have developed a new data-driven framework, called Stimulus Information Representation (SIR), that enables us to better understand how the brain processes information from measures of brain activity and behavioral responses. In this article, we explain this approach, its strengths and limitations, and how it can be applied to understand how the brain processes information to perform behavior in a task.Much of human cognition starts with a brain that categorizes stimulus information to behave adaptively 1-3 . Thus, human brains are compulsive categorizers that use stimulus information to perform different categorization tasks. Consider the street scene shown on the left-hand side of Figure 1. The brain can perform numerous categorization tasks on this image: it can identify the country, city, and street; the houses and shops; the moving and stationary cars, their makes and models, age and condition; the people shopping or those just passing by, as well as their gait, identity, emotion and social interactions; it can also infer the weather, time of day, season, and so on.So, when we record brain activity, we need to circumscribe an experimental task in order to know which task the brain has performed when we analyze its activity, even when a task involves just a single image. And when we circumscribe a task, we still need to characterize the stimulus information that supports a particular categorization. Otherwise, we will not know what information the brain has processed when it categorized the street, or the make of a car, or the identity of a face or its expression, from the same image.Such task-relevant information processing is a generic, but often neglected theoretical point, that applies both to the interpretation of any sensory categorization in the brain and to its models. For example, Convolutional Neural Networks 4,5 (CNNs) are braininspired, hierarchically organized network models that also use stimulus information to perform multiple categorization tasks, with apparent human-like capabilities. But to realize the promise of CNNs as models of brain information processing 6-12 , a neglected pre-condition needs to be met -that these models ...

Perception-driven dynamics of mimicry based on attractor field model

“…Moreover, the distance-based multidimensional similarity space, proposed by Valentine [74,75] and briefly discussed in this study, has already been subjected to experimental quantification using functional magnetic resonance imaging [97]. Now, when high-throughput brain activity data are becoming widely available, we need approaches that would connect these data with a biological theory of communication and deception [98,99]. And finally, in recent years, we saw the first studies which use methods connecting animal behaviour with specific neural dynamics [65,67,100].…”

Grounding deep neural network predictions of human categorization behavior in understandable functional features: The case of face identity

“…Behaviour DNN Figure 1: Trivariate relationship to understand the functional features of DNN models that predict human behaviour. This Venn diagram shows how the understandable part of DNN model predictions of human responses can be formalised as the intersection (shown in white) of information about behaviour from a generative model of the stimulus and information about behaviour from a DNN model (Schyns & Ince, 2019;Schyns et al, 2020). The visual input (shown in blue) is processed in an unknown and inaccessible way in the brain to elicit human behaviour (shown in green).…”

“…The typical assessment of a candidate DNN model evaluates how it informs human responses, by computing the bivariate relationship (the yellow set intersection) between human responses (the green set) and DNN predictions (the red set). Here, we instead evaluated the triple white set intersection (Schyns & Ince, 2019;Schyns et al, 2020) between human behavioural responses, DNN predictions and the experimentally controlled GMF features (the blue set). We then detailed how each candidate DNN model represents the GMF features to predict human behaviour.…”