Searching for Rewards Like a Child Means Less Generalization and More Directed Exploration

Garvert

et al. 2020

Preprint

Self Cite

There is a resurgence of interest in "cognitive maps" based on recent evidence that the hippocampal-entorhinal system encodes both spatial and non-spatial information, with far-reaching implications for human behavior. Yet little is known about the commonalities and differences in the computational principles underlying human learning and decision making in spatial and non-spatial domains. We use a within-subject design to examine how humans search for either spatially or conceptually correlated rewards. Using a Bayesian learning model, we find evidence for the same computational mechanisms of generalization across domains. While participants were sensitive to expected rewards and uncertainty in both tasks, how they leveraged this knowledge to guide exploration was different: participants displayed less uncertainty-directed and more random exploration in the conceptual domain. Moreover, experience with the spatial task improved conceptual performance, but not vice versa. These results provide important insights about the degree of overlap between spatial and conceptual cognition. 2 the world using spatial metaphors [4, 5], and commonly use concepts like "space" or 3 "distance" in mathematical descriptions of abstract phenomena. 4In line with these observations, previous theories have argued that reasoning about 5 abstract conceptual information follows the same computational principles as spatial 6 1/36 reasoning [6][7][8]. This has recently gained new support from neuroscientific evidence 7 suggesting that common neural substrates are the basis for knowledge representation 8 across domains [9][10][11][12][13]. 9 One important implication of these accounts is that reinforcement learning [14] in 10 non-spatial domains may rely on a map-like organization of information, supported by 11 the computation of distances or similarities between experiences. Here, we ask to what 12 extent does the search for rewards depend on the same distance-dependent 13 generalization across domains? We formalize a computational model that incorporates 14 distance-dependent generalization and test it in a within-subject experiment, where 15 either spatial features or abstract conceptual features are predictive of rewards. This 16 allows us to study learning, decision making, and exploration in spatial versus 17 conceptual domains, in order to gain insights into the organizational structure of 18 cognitive representations in both domains. 19Whereas early psychological theories described reinforcement learning as merely 20 developing an association between stimuli, responses and rewards [15][16][17], more recent 21 studies have recognized that the structure of representations plays an important role in 22 making value-based decisions [11, 18] and is particularly important for knowing how to 23 generalize from limited data to novel situations [19, 20]. This idea dates back to Tolman, 24 who famously argued that both rats and humans extract a "cognitive map" of the 25 environment [21]. This cognitive map encodes relationships between e...

Section: Parameter Estimatessupporting

confidence: 81%

Section: Uncertainty-directed Explorationmentioning

confidence: 99%

Similarities and differences in spatial and non-spatial cognitive maps

Garvert

et al. 2020

Preprint

Self Cite

Section: Computational Models Of Learning Generalization and Searchmentioning

confidence: 99%

Similarities and differences in spatial and non-spatial cognitive maps

Garvert

et al. 2020

PLoS Comput Biol

Self Cite

Learning and generalization in spatial domains is often thought to rely on a "cognitive map", representing relationships between spatial locations. Recent research suggests that this same neural machinery is also recruited for reasoning about more abstract, conceptual forms of knowledge. Yet, to what extent do spatial and conceptual reasoning share common computational principles, and what are the implications for behavior? Using a within-subject design we studied how participants used spatial or conceptual distances to generalize and search for correlated rewards in successive multi-armed bandit tasks. Participant behavior indicated sensitivity to both spatial and conceptual distance, and was best captured using a Bayesian model of generalization that formalized distance-dependent generalization and uncertainty-guided exploration as a Gaussian Process regression with a radial basis function kernel. The same Gaussian Process model best captured human search decisions and judgments in both domains, and could simulate realistic learning curves, where we found equivalent levels of generalization in spatial and conceptual tasks. At the same time, we also find characteristic differences between domains. Relative to the spatial domain, participants showed reduced levels of uncertainty-directed exploration and increased levels of random exploration in the conceptual domain. Participants also displayed a one-directional transfer effect, where experience in the spatial task boosted performance in the conceptual task, but not vice versa. While confidence judgments indicated that participants were sensitive to the uncertainty of their knowledge in both tasks, they did not or could not leverage their estimates of uncertainty to guide exploration in the conceptual task. These results support the notion that value-guided learning and generalization recruit cognitive-map dependent computational mechanisms in spatial and conceptual domains. Yet both behavioral and model-based analyses suggest domain specific differences in how these representations map onto actions.

“…The subway passenger cover story was used to provide intuitions about graph correlated functions, similar to our example from the introduction. The color aid was generated through a continuous, linear mapping (similar to Meder et al, 2020;Schulz, Wu, Ruggeri, & Meder, 2019;, with both hue and brightness changing monotonically with value. Additionally, participants observed 10 fully revealed graphs to familiarize themselves with the task and completed a comprehension check before starting the task.…”

Section: Methodsmentioning

confidence: 99%

Inference and search on graph-structured spaces

Gershman³

2020

Preprint

Self Cite

How do people learn functions on structured spaces? And how do they use this knowledge to guide their search for rewards in situations where the number of options is large? We study human behavior on structures with graph-correlated values and propose a Bayesian model of function learning to describe and predict their behavior. Across two experiments, one assessing function learning and one assessing the search for rewards, we find that our model captures human predictions and sampling behavior better than several alternatives, generates human-like learning curves, and also captures participants' confidence judgements. Our results extend past models of human function learning to more complex, graph-structured domains.