Computational reconstruction of mental representations using human behavior

Caplette, Laurent; Turk‐Browne, Nicholas B.

doi:10.31234/osf.io/7fdvw

Cited by 3 publications

(3 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All raw and preprocessed data generated and analyzed during this study are available at https://osf.io/mp3s6/ 102 . The following publicly available data were also used in the study: Behavioral dataset on semantic word arrangement (https://osf.io/um3qg/) 50 ; Visual Genome dataset (https://homes.cs.washington.edu/~ranjay/visualgenome/ index.html) 48 ; GloVe word embedding (https://nlp.stanford.edu/ projects/glove/) 67 ; pretrained adversarially robust ResNet-50 (https:// github.com/MadryLab/robust_representations) 89 ; and ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) 2012 dataset (https://www.image-net.org/challenges/LSVRC/2012/) 58 .…”

Section: Reporting Summarymentioning

confidence: 99%

Computational reconstruction of mental representations using human behavior

Caplette,

Turk-Browne

2024

Nat Commun

View full text Add to dashboard Cite

Revealing how the mind represents information is a longstanding goal of cognitive science. However, there is currently no framework for reconstructing the broad range of mental representations that humans possess. Here, we ask participants to indicate what they perceive in images made of random visual features in a deep neural network. We then infer associations between the semantic features of their responses and the visual features of the images. This allows us to reconstruct the mental representations of multiple visual concepts, both those supplied by participants and other concepts extrapolated from the same semantic space. We validate these reconstructions in separate participants and further generalize our approach to predict behavior for new stimuli and in a new task. Finally, we reconstruct the mental representations of individual observers and of a neural network. This framework enables a large-scale investigation of conceptual representations.

show abstract

Section: Reporting Summarymentioning

confidence: 99%

Computational reconstruction of mental representations using human behavior

Caplette,

Turk-Browne

2024

Nat Commun

View full text Add to dashboard Cite

show abstract

“…Attending not just to sampleaveraged representations as currently practiced but to the heterogeneity of visual features used by different people for the same category within and across contexts is critical to not reify the average representation as a generalizable racialization that operates for everyone, but to instead map topographies of racializing processes. There is always a danger in face studies of treating the face as an isolated and determining component of race classifications (which can biologize race), therefore the development of models that attend to wider kinds of visible features could help identify different material inputs to racialization (e.g., Caplette & Turk-Browne, 2022).…”

Section: Minimizing Racecraftmentioning

confidence: 99%

Facecraft: Race Reification in Psychological Research With Faces

Martinez

2023

Perspect Psychol Sci

View full text Add to dashboard Cite

Faces are socially important surfaces of the body on which various meanings are attached. The widespread physiognomic belief that faces inherently contain socially predictive value is why they make a generative stimulus for perception research. However, critical problems arise in studies that simultaneously investigate faces and race. Researchers studying race and racism inadvertently engage in various research practices that transform faces with specific phenotypes into straightforward representatives of their presumed race category, thereby taking race and its phenotypic associations for granted. I argue that research practices that map race categories onto faces using bioessentialist ideas of racial phenotypes constitute a form of racecraft ideology, the dubious reasoning of which presupposes the reality of race and mystifies the causal relation between race and racism. In considering how to study racism without reifying race in face studies, this article places these practices in context, describes how they reproduce racecraft ideology and impair theoretical inferences, and then suggests counterpractices for minimizing this problem.

show abstract

“…Mental representations serve as a substrate for a variety of cognitive tasks such as decision-making, communication and memory (Anderson, 1990). Understanding the structure of those representation is a core problem in cognitive science and is the subject of a large corpus of work in the psychological literature (Shepard, 1980(Shepard, , 1987Ghirlanda & Enquist, 2003;Battleday, Peterson, & Griffiths, 2020;Peterson, Abbott, & Griffiths, 2018;Jha, Peterson, & Griffiths, 2020;Caplette & Turk-Browne, 2022;Hebart, Zheng, Pereira, & Baker, 2020).…”

Section: Introductionmentioning

confidence: 99%

Predicting Human Similarity Judgments Using Large Language Models

Marjieh¹,

Sucholutsky²,

Sumers³

et al. 2022

Preprint

View full text Add to dashboard Cite

Similarity judgments provide a well-established method for accessing mental representations, with applications in psychology, neuroscience and machine learning. However, collecting similarity judgments can be prohibitively expensive for naturalistic datasets as the number of comparisons grows quadratically in the number of stimuli. One way to tackle this problem is to construct approximation procedures that rely on more accessible proxies for predicting similarity. Here we leverage recent advances in language models and online recruitment, proposing an efficient domain-general procedure for predicting human similarity judgments based on text descriptions. Intuitively, similar stimuli are likely to evoke similar descriptions, allowing us to use description similarity to predict pairwise similarity judgments. Crucially, the number of descriptions required grows only linearly with the number of stimuli, drastically reducing the amount of data required. We test this procedure on six datasets of naturalistic images and show that our models outperform previous approaches based on visual information.

show abstract

Computational reconstruction of mental representations using human behavior

Cited by 3 publications

References 54 publications

Computational reconstruction of mental representations using human behavior

Computational reconstruction of mental representations using human behavior

Facecraft: Race Reification in Psychological Research With Faces

Predicting Human Similarity Judgments Using Large Language Models

Contact Info

Product

Resources

About