BOLD Moments: modeling short visual events through a video fMRI dataset and metadata

Lahner, Benjamin; Dwivedi, Kshitij; Iamshchinina, Polina; Graumann, Monika; Lascelles, Alex; Roig, Gemma; Gifford, Alessandro Thomas; Pan, Bowen; Jin, SouYoung; Murty, N. Apurva Ratan; Kay, Kendrick; Oliva, Aude; Cichy, Radoslaw Martin

doi:10.1101/2023.03.12.530887

Cited by 6 publications

(6 citation statements)

References 210 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The stimulus set is a subset of the MiT dataset 16 , which has been used in other neuroimaging studies 1,54 . We procedurally removed videos to select a set reasonable for a cognitive neuroscience experiment (n = 250) based on many factors including the resolution and depiction of everyday actions based on the American Time Use Survey 17 .…”

Section: Stimulus Set Videosmentioning

confidence: 99%

Hierarchical organization of social action features along the lateral visual pathway

McMahon¹,

Bonner²,

Işık³

2023

Preprint

View full text Add to dashboard Cite

Recent theoretical work has argued that in addition to the classical ventral (what) and dorsal (where/how) visual streams, there is a third visual stream on the lateral surface of the brain specialized for processing social information. Like visual representations in the ventral and dorsal streams, representations in the lateral stream are thought to be hierarchically organized. However, no prior studies have comprehensively investigated the organization of naturalistic, social visual content in the lateral stream. Here we used a data-rich approach to test the hypothesis that social action information is extracted hierarchically along this pathway. Social scenes pose several computational challenges beyond those of object recognition: they are often highly dynamic and involve relations between two or more people. To address these challenges, we curated a naturalistic stimulus set of 250 three-second videos of two people engaged in everyday actions. Each clip was richly annotated for its low-level visual features, mid-level scene & object properties, visual social primitives (including the distance between people and the extent to which they were facing), and high-level information about social interactions and affective content. Using a condition-rich fMRI experiment and a within-subject encoding model approach, we found that low-level visual features are represented in early visual cortex (EVC) and middle temporal area (MT), mid-level visual-social features in extrastriate body area (EBA) and lateral occipital complex (LOC), and high-level social interaction information along the superior temporal sulcus (STS). Communicative interactions, in particular, explained unique variance in anterior regions of the STS, even when accounting for variance explained by all other labeled features. Taken together, these results provide support for representation of increasingly abstract social visual content—consistent with hierarchical organization—along the lateral visual stream and suggest that recognizing communicative actions may be a key computational goal of the STS and lateral pathway.

show abstract

Section: Stimulus Set Videosmentioning

confidence: 99%

Hierarchical organization of social action features along the lateral visual pathway

McMahon¹,

Bonner²,

Işık³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Excitingly, there are new hardcore fMRI initiatives on the horizon where the experiment moves beyond perception of static images and instead uses dynamic videos [35] or interactive closed-loop behavioral protocols, such as game-play [36]. These and other initiatives will expand hardcore fMRI beyond the scope of sensory neuroscience into a wide range of topics in cognitive neuroscience.…”

Section: How Is Hardcore Fmri Advancing Neuroscience?mentioning

confidence: 99%

Hardcore fMRI: amplifying the quality and scale of fMRI data to learn new insights into human brain function

Kupers,

Knapen,

Merriam

et al. 2024

Preprint

View full text Add to dashboard Cite

Neuroscience has witnessed the acquisition and open sharing of massive fMRI datasets that place intense emphasis on data quality and quantity in individual participants. These datasets strive for extensive sampling of cognitive phenomena. In contrast to conventional fMRI studies where data are typically noisy and sample a small number of experimental conditions, this new "hardcore" approach to fMRI enables a rich hypothesis space and supports detailed investigation of computational mechanisms at the single- voxel level. Hardcore fMRI has already sparked exciting discoveries in neuroscience and machine learning which would have been difficult to achieve with conventional fMRI. We envision a future in which the proliferation of hardcore fMRI datasets establishes a new collaborative scientific ecosystem that provides unprecedented opportunities for deciphering brain function.

show abstract

“…Here, we take the latter approach, but rather than aiming to understand static scene responses in the ventral visual stream, which are well modeled by most current image models [24], we aim to understand dynamic visual responses across the brain, focusing on the lateral visual stream. In addition to benchmarking vision models, we also use language models to predict behavioral ratings and visually-evoked neural responses as has been done in prior work with static scenes [27,28,29,30,31,32]. As in Conwell et al [27], we use multiple language models as predictors and selectively perturb the sentence captions of the stimuli to provide insights into the kind of linguistic features that are predictive of visual responses.…”

Section: Related Workmentioning

confidence: 99%

“…As in Conwell et al [27], we use multiple language models as predictors and selectively perturb the sentence captions of the stimuli to provide insights into the kind of linguistic features that are predictive of visual responses. While modeling dynamic visual events is a growing area of interest [32], this is the first investigation of benchmarking many models in response to naturalistic videos of human actions.…”

Section: Related Workmentioning

confidence: 99%

“…A limitation of the current study is that the human dataset we modeled here only contained behavioral and neural responses to 250 naturalistic video clips. While larger dynamic neuroscience datasets do exist [32], they are not yet publicly available and do not rival existing static image datasets [62]. Likewise, computer vision video datasets [63,64,65] lack the size and diversity of image datasets [66,65].…”

Section: Limitationsmentioning

confidence: 99%

See 1 more Smart Citation

Modeling dynamic social vision highlights gaps between deep learning and humans

Garcia,

McMahon,

Conwell

et al. 2024

Preprint

View full text Add to dashboard Cite

Deep learning models trained on computer vision tasks are widely considered the most successful models of human vision to date. The majority of work that supports this idea evaluates how accurately these models predict brain and behavioral responses to static images of objects and natural scenes. Real-world vision, however, is highly dynamic, and far less work has focused on evaluating the accuracy of deep learning models in predicting responses to stimuli that move, and that involve more complicated, higher-order phenomena like social interactions. Here, we present a dataset of natural videos and captions involving complex multi-agent interactions, and we benchmark 350+ image, video, and language models on behavioral and neural responses to the videos. As with prior work, we find that many vision models reach the noise ceiling in predicting visual scene features and responses along the ventral visual stream (often considered the primary neural substrate of object and scene recognition). In contrast, image models poorly predict human action and social interaction ratings and neural responses in the lateral stream (a neural pathway increasingly theorized as specializing in dynamic, social vision). Language models (given human sentence captions of the videos) predict action and social ratings better than either image or video models, but they still perform poorly at predicting neural responses in the lateral stream. Together these results identify a major gap in AI's ability to match human social vision and highlight the importance of studying vision in dynamic, natural contexts.

show abstract

BOLD Moments: modeling short visual events through a video fMRI dataset and metadata

Cited by 6 publications

References 210 publications

Hierarchical organization of social action features along the lateral visual pathway

Hierarchical organization of social action features along the lateral visual pathway

Hardcore fMRI: amplifying the quality and scale of fMRI data to learn new insights into human brain function

Modeling dynamic social vision highlights gaps between deep learning and humans

Contact Info

Product

Resources

About