THINGS-data: A multimodal collection of large-scale datasets for investigating object representations in human brain and behavior

Abstract

Understanding object representations requires a broad, comprehensive sampling of the objects in our visual world with dense measurements of brain activity and behavior. Here we present THINGS-data, a multimodal collection of large-scale datasets comprising functional MRI, magnetoencephalographic recordings, and 4.70 million similarity judgments in response to thousands of photographic images for up to 1,854 object concepts. THINGS-data is unique in its breadth of richly-annotated objects, allowing for testing countless hypotheses at scale while assessing the reproducibility of previous findings. Beyond the unique insights promised by each individual dataset, the multimodality of THINGS-data allows combining datasets for a much broader view into object processing than previously possible. Our analyses demonstrate the high quality of the datasets and provide five examples of hypothesis-driven and data-driven applications. THINGS-data constitutes the core release of the THINGS initiative (https://things-initiative.org) for bridging the gap between disciplines and the advancement of cognitive neuroscience.

Visual object topographic motifs emerge from self-organization of a unified representational space

“…Recent open, large-scale condition-rich fMRI datasets are now available (e.g. NSD dataset, Allen et al, 2022; THINGS dataset, Hebart et al, 2019, 2022) which can enable the development of cortical topographic metrics beyond these macro- and meso-scale signatures probed for here. Thus, going forward, there is clear work to do towards mapping these computational models more directly to the cortex (c.f.…”

“…Recent open, large-scale condition-rich fMRI datasets are now available (e.g. NSD dataset ( 104 ); THINGS dataset ( 105, 106 )) which can enable the development of cortical topographic metrics beyond these macro- and meso-scale signatures probed for here. Thus, going forward, there is clear work to do towards mapping these computational models more directly to the cortex ( 49 ), and assessing how they succeed and fail at capturing the systematic response structure to thousands of natural images across the cortical surface.…”

Visual Representations: Insights from Neural Decoding

“…Increasing the variation within each category serves to dampen the low-level confounds that might otherwise dominate measurements of neural representations . Studying object-specific neural responses with such large stimulus sets has the advantage of more closely mimicking natural vision, as well as allowing more fine-grained analyses of visual features, categories, and semantics (Chang et al 2019, Hebart et al 2023. For example, models of image statistics and object category can be compared with the neural data to assess how much each model accounts for the variance in neural information (Grootswagers et al 2019a, Moerel et al 2022a).…”

How well do rudimentary plasticity rules predict adult visual object learning?

“…However, while empirical behavioral benchmarks exist for visual tasks involving known object categories (18–20), the field currently lacks a publicly-available set of benchmarks for comparing models to humans as they learn novel object categories, making it difficult to gauge progress in the field.…”

“…While behavioral benchmarks exist for visual tasks involving known object categories (e.g. Rajalingham et al(2018 ); Geirhos et al (2021 ); Hebart et al (2022 )), the field currently lacks a publicly-available set of benchmarks for comparing models to humans for learning tasks involving novel objects, making it diffcult to gauge progress in the field.…”