The error in total error reduction

Abstract

Most models of human and animal learning assume that learning is proportional to the discrepancy between a delivered outcome and the outcome predicted by all cues present during that trial (i.e., total error across a stimulus compound). This total error reduction (TER) view has been implemented in connectionist and artificial neural network models to describe the conditions under which weights between units change. Electrophysiological work has revealed that the activity of dopamine neurons is correlated with the total error signal in models of reward learning. Similar neural mechanisms presumably support fear conditioning, human contingency learning, and other types of learning. Using a computational modelling approach, we compared several TER models of associative learning to an alternative model that rejects the TER assumption in favor of local error reduction (LER), which assumes that learning about each cue is proportional to the discrepancy between the delivered outcome and the outcome predicted by that specific cue on that trial. The LER model provided a better fit to the reviewed data than the TER models. Given the superiority of the LER model with the present data sets, acceptance of TER should be tempered.

The functions of contexts in associative learning

“…Presumably context continues to acquire associative strength with the US after the discrete cue has reached an asymptote because the context has a lower salience than the discrete cue. Moreover, memory of the training context is thought to negatively interact with discrete cues at the time of test (Stout & Miller, 2007; also see Witnauer, Urcelay, & Miller, 2013, for simulations). In summary, the various explanations of the PPD all emphasize a role for the context, but differ in terms of what this role is, with some arguing that it reflects a modulatory function whereas others argue for a cueing function.…”

Methods of comparing associative models and an application to retrospective revaluation

“…Such revisions to empirical benchmarks should be informed by formal modeling procedures as those procedures are best able to reveal the empirical effects that have the potential to differentiate among models. We have used retrospective revaluation (e.g., Kaufman & Bolles, 1981) and counteraction effects (Wheeler & Miller, 2008) to differentiate among models (e.g., Witnauer & Miller, 2011; Witnauer, Urcelay, & Miller, 2014). However, formal, simulation-based comparisons between models are almost absent from the animal learning literature (e.g., Kutlu & Schmajuk, 2012).…”

“…Formal model comparison is the most appropriate strategy for differentiating among models when multiple models are able to explain some empirical benchmark. For example, most associative models predict blocking but formal model comparisons have revealed that only some associative models are able to accurately predict the degree of blocking (usually incomplete) that is observed in most experiments (e.g., Witnauer et al, 2014). Moreover, even when reasonable empirical criteria seem to differentiate between models, researchers can underestimate the flexibility of models, which might lead to erroneous rejection of a potentially viable model.…”

“…For example, the Rescorla-Wagner model fails to explain latent inhibition (Lubow & Moore, 1959), spontaneous recovery from extinction (Pavlov, 1929), retrospective revaluation (Kaufman & Bolles, 1981), taste-odor potentiation (Rusiniak, Hankins, Garcia, & Brett, 1979), and backward blocking (Shanks, 1985; for reviews, see Miller, Barnett, & Grahame, 1995; Witnauer, Urcelay, & Miller, 2014). Additionally, the Rescorla-Wagner model does not explain cue facilitation effects such as potentiation or second-order conditioning.…”

Retrospective revaluation: The phenomenon and its theoretical implications

“…In recent years, there has been wide acceptance of the principle of total error reduction, an acceptance that in our view has been at least partially driven by a bandwagon effect. This has occurred despite considerable data inconsistent with it (see Witnauer, Urcelay, and Miller, 2014, for a review). Perhaps most notable among the behavioral observations challenging the total error reduction principle is Rescorla’s (2000, 2001, 2008) finding that two cues matched in associability (through counterbalancing) that are reinforced together in compound (i.e., same total error) change differentially as a function of their different associative strengths going into that trial.…”