The rat approximates an ideal detector of changes in rates of reward: Implications for the law of effect.

Abstract

Rats responded on 2 levers delivering brain stimulation reward on concurrent variable interval schedules. Following many successive sessions with unchanging relative rates of reward, subjects adjusted to an eventual change slowly and showed spontaneous reversions at the beginning of subsequent sessions. When changes in rates of reward occurred between and within every session, subjects adjusted to them about as rapidly as they could in principle do so, as shown by comparison to a Bayesian model of an ideal detector. This and other features of the adjustments to frequent changes imply that the behavioral effect of reinforcement depends on the subject's perception of incomes and changes in incomes rather than on the strengthening and weakening of behaviors in accord with their past effects or expected results. Models for the process by which perceived incomes determine stay durations and for the process that detects changes in rates are developed.

Acquisition and extinction in autoshaping.

“…Although our model's focus is on acquisition and extinction, it is governed by a rich set of quantitative constraints. First, because animals can be ideal detectors of rates in some circumstances (e.g., Gallistel, Mark, King, & Latham, 2001), we required an account under which their acquisition of responding could be given a rational statistical basis. In this sense, the rate of responding during learning should be based on an optimal evaluation of the contingencies given the observations so far provided.…”

“…Normative models suggest that the decisions of the animals to start and stop responding are actually correct according to a well-specified statistical model, given the information they have received about the relationship between stimuli and rewards, and prior expectations. Gallistel, Mark, King, and Latham (2001) showed directly that animals can be statistically optimal detectors of rate changes such as those that underlie autoshaping. Gallistel and Gibbon (2000) suggested a model for acquisition and extinction data that they called rate estimation theory (RET).…”

The Neural Mechanisms that Underlie Decision Making

“…Thus, changes in income are apparent much sooner than are changes in profit. We showed that the adjustment to a new ratio of incomes often occurred before there was any evidence of a change in the relative profitability of visits to the two locations ( Gallistel et al ., 2001 ).…”

“…Traditionally, models have assumed that matching was the slow equilibration of profit by a hill-climbing process in which the investment ratio was adjusted until the profit ratio was 1:1 ( Herrnstein and Vaughan, 1980 ). We have shown, however, that the process of adjustment can be extremely fast; it can go to completion within the span of a single interreinforcement interval at one of the locations ( Gallistel et al ., 2001 ). Moreover, when computed on a visit-byvisit basis, profit is a much noisier variable (because income appears in its denominator, and its numerator, visit duration, is an exponentially distributed random variable).…”

“…They are not making the choice they appear to be making. Rather, they are deciding when to abandon whichever option they have already chosen ( Belke, 1992 ;Gibbon, 1995 ;Gallistel and Gibbon, 2000 ;Gallistel et al ., 2001 ).…”