Implicit Value Updating Explains Transitive Inference Performance: The Betasort Model

Abstract

Transitive inference (the ability to infer that B > D given that B > C and C > D) is a widespread characteristic of serial learning, observed in dozens of species. Despite these robust behavioral effects, reinforcement learning models reliant on reward prediction error or associative strength routinely fail to perform these inferences. We propose an algorithm called betasort, inspired by cognitive processes, which performs transitive inference at low computational cost. This is accomplished by (1) representing stimulus positions along a unit span using beta distributions, (2) treating positive and negative feedback asymmetrically, and (3) updating the position of every stimulus during every trial, whether that stimulus was visible or not. Performance was compared for rhesus macaques, humans, and the betasort algorithm, as well as Q-learning, an established reward-prediction error (RPE) model. Of these, only Q-learning failed to respond above chance during critical test trials. Betasort’s success (when compared to RPE models) and its computational efficiency (when compared to full Markov decision process implementations) suggests that the study of reinforcement learning in organisms will be best served by a feature-driven approach to comparing formal models.

The Development of Size Sequencing Skills: An Empirical and Computational Analysis

“… action selection is widely used in cognitive modeling. Jensen et al () used it within one of their transitive inference simulations as an action selection routine. Also, we represent each aspect of the agent's memory (WM and LTM) to inform action selection here, which seems intuitively plausible and faithful.…”

“…It defines a set of probability distributions that are all changed according to a Bayesian inference procedure on every stimulus selection, resulting in a shift of their probability mass along the unit (0–1) interval and ultimately a ranking. The way we carry out this Bayesian inference is similar to the algorithm used by Jensen et al (), but much simpler. Their study compares binary choice data from both rhesus macaque ( Macaca mulatta ) and human adult participants to binary choice model data.…”

“…They concede, as we do, that there is no biological evidence for all neural representations of a stimulus set being updated on stimulus selection; the updating of all the distributions across the set is thus not a standard way of representing learning in cognitive models. As Jensen et al () argue, the implicit updating of all stimuli in a set via a plausible Bayesian inference routine is a hypothesis to be falsified by neuroscientific data. In support of this, we note that the collective updating distributions, as per Equations (4a) and (4b), is mathematically necessary for them to form a rank.…”

“…Our model does three things to help our understanding here. First, it infers the order of a discrete set of size items by trial and error, via a Bayesian ranking mechanism similar to that of Jensen, Muñoz, Alkan, Ferrera, and Terrace () that creates a ranked representation in LTM. Second, it makes explicit the transition to spontaneous ordering of size‐related sequences.…”

“…Our model does several things to help our understanding of how sequential size understanding emerges. First, simulating the 5‐year‐old child, the model infers the order of a set of shapes by trial and error, via a Bayesian ranking algorithm similar to Jensen et al (). This process creates a gradually more precisely ranked representation of the items in LTM.…”