Asymmetric reinforcement learning facilitates human inference of transitive relations

Abstract

Humans and other animals are capable of inferring never-experienced relations (for example, A > C) from other relational observations (for example, A > B and B > C). The processes behind such transitive inference are subject to intense research. Here we demonstrate a new aspect of relational learning, building on previous evidence that transitive inference can be accomplished through simple reinforcement learning mechanisms. We show in simulations that inference of novel relations benefits from an asymmetric learning policy, where observers update only their belief about the winner (or loser) in a pair. Across four experiments (n = 145), we find substantial empirical support for such asymmetries in inferential learning. The learning policy favoured by our simulations and experiments gives rise to a compression of values that is routinely observed in psychophysics and behavioural economics. In other words, a seemingly biased learning strategy that yields well-known cognitive distortions can be beneficial for transitive inferential judgements.

Implicit Counterfactual Effect in Partial Feedback Reinforcement Learning: Behavioral and Modeling Approach

“…In the model recovery analysis, our aim is to investigate whether the models in the model space can be distinguished from each other. To do this, we used the model recovery approach in the paper of Wilson and Collins (Wilson and Collins, 2019 ; Ciranka et al, 2022 ). According to this approach we calculated two metrics: the conditional probability that a model fits best given the true generative model [ p ( fit | gen )], and the conditional probability that the data was generated by a specific model, given it is the best fitted model [ p ( gen | fit )].…”

“…Unfortunately, some of the models in our model space have rather similar behavior on this task (e.g., the Hybrid model with w = 1 is identical to the SQL model), therefore we have large off-diagonal elements in this matrix ( Figure 8 ). Since the model recovery was not perfect, we conducted p ( gen | fit ) analysis, which is a more critical metric to investigate model recovery analysis (Wilson and Collins, 2019 ; Ciranka et al, 2022 ). As can be seen in the Figure 8 , in the Partial version, all diagonal entries of the p ( gen | fit ) matrix, except OL 2 are dominant in their columns which shows that all the models except OL 2 could be identified well.…”

EEG-representational geometries and psychometric distortions in approximate numerical judgment

“…Such subjective “compression” of magnitude can be observed in a great variety of settings, from basic sensory-perceptual judgments (Fechner, 1860; Stevens, 1957) to economic decisions (Kellen et al, 2016; McAllister & Tarbert, 1999; Tversky & Kahneman, 1992). There exist various theoretical accounts for the origin and potential benefits of subjective compression in perception and decision making (Bhui & Gershman, 2018; Ciranka et al, 2022; de Gardelle & Summerfield, 2011; Li et al, 2017; Pardo-Vazquez et al, 2019; Stewart et al, 2006; Summerfield & Li, 2018; Vandormael et al, 2017). Our present results seem to contrast these vast literatures with the finding that human brain signals tended to reflect the magnitude of numerical values in an anti -compressed fashion, even when they were compressed in later choice.…”

Dynamic emergence of relational structure network in human brains

“…Hippocampus is found to be involved in the process (Schapiro et al, 2013; Schapiro et al, 2016), and interestingly, the hexagonal coding properties of grid cells correspond to the eigenvectors of transition probability matrix in spatial maps (Stachenfeld et al, 2017). Importantly, in addition to the simple one-dimensional transition structure (Ciranka et al, 2022; Liu, Dolan, Kurth-Nelson, & Behrens, 2019; Nelli et al, 2021), events could also be linked in a graph-like network with different structures, e.g., lattice, hexagonal, community, hierarchical tree, etc (Kahn et al, 2018; Kemp & Tenenbaum, 2008; Lynn et al, 2020; Mark et al, 2020; Schapiro et al, 2013). Consistent with previous graph-learning studies (Kahn et al, 2018; Lynn et al, 2020), we demonstrate that human subjects could learn the relationship between images embedded in a transition network.…”

“…What is the computational mechanism underlying the formation of a higher-order community structure? Computational models of associative learning and reinforcement learning have been previously used to explain transition probability learning (Ciranka et al, 2022; Maheu et al, 2019), and the successor representation (SR) model could account for the emergence of community structure (Pudhiyidath et al, 2021; Stachenfeld et al, 2017). Here we built three computational models to seek the computational nature of the structural network.…”