A Computational Model of Learned Avoidance Behavior in a One-Way Avoidance Experiment

Abstract

Learned avoidance behavior is a critical component of animal survival; thus, any model of animal learning must account for the phenomenon. The two dominant theories of animal behavior during the early 20th century were Pavlovian conditioning (Pavlov, 1927) and Thorndikian law of effect (Thorndike, 1898). Pavlov proposed that all animal learning could be explained by conditioning (also referred to as stimulus substitution). Thorndike's law of effect was founded on response substitution. Each theory explained an important characteristic of behavior: anticipation in the case of Pavlovian conditioning or goal-directed behavior in the case of Thorndikian law of effect. However, neither theory could account for the entirety of animal behavior. Miller and Konorski (1928/1969), and eventually others, proposed that an explanation of animal behavior required not one theory or the other, but both, a two-process theory of learning. Many view Mowrer's (1947) two-factor theory of learning as the definitive two-process theory. According to this theory, the first process involves fear being classically conditioned to the conditioned stimulus (CS) associated with the punishment signal. That CS is called the warning signal because it reliably precedes the punishment signal and is learned to be a cue for the punishment signal. The animal's conditioned response (CR) to the warning signal, fear,

A temporal difference account of avoidance learning

“…There is also a developing understanding at all these levels of the relationship between habits and motivationally sophisticated goal-directed actions. By contrast, despite some notable studies (Grossberg 1972;Schmajuk and Zanutto 1997;Daw et al 2002;Johnson et al 2002;Seymour et al 2004), the functional bases of aversive learning remain more obscure.…”

Two-factor theory, the actor--critic model, and conditioned avoidance

“…If the agent performs action a in state s and things turn out better than it expected, it should strengthen the association between s and a-that is, increase p(s,a). If things turn out worse one presented here is that by Johnson et al (2002), which was based on Klopf, Morgan, and Weaver's (1993) associative control process (ACP) framework. The ACP framework is closely related to two-factor theory (Klopf et al, 1993) and to the actor-critic (Johnson et al, 2002), but the use of standard reinforcement learning machinery in the present article has the advantage of linking more directly to the extensive work on reinforcement learning models of conditioning and to the well-developed computational theory of reinforcement learning.…”

“…If things turn out worse one presented here is that by Johnson et al (2002), which was based on Klopf, Morgan, and Weaver's (1993) associative control process (ACP) framework. The ACP framework is closely related to two-factor theory (Klopf et al, 1993) and to the actor-critic (Johnson et al, 2002), but the use of standard reinforcement learning machinery in the present article has the advantage of linking more directly to the extensive work on reinforcement learning models of conditioning and to the well-developed computational theory of reinforcement learning. The models of Grossberg (1972) and Schmajuk and Zanutto (1997), while capturing several empirical findings and being sources of important insights, were more exclusively motivated by the behavioral findings, without independent computational motivation, and without connecting to the known neural bases of conditioning.…”

“…A common approach to keeping track of time is to use a tapped delay line (Daw, Courville, & Touretzky, 2006;Daw & Touretzky, 2002;Johnson et al, 2002;Montague, Dayan, & Sejnowski, 1996;Schultz, Dayan, & Montague, 1997;Smith et al, 2004;Sutton & Barto, 1990). The idea is to "unfold" a state into multiple substates that represent the passage of time within the original state.…”

“…In addition, their main emphasis was on the effects of dopaminergic manipulations, -whereas the emphasis here is on the relation of the actorcritic to two-factor theory. Other computational models of avoidance learning that bear close resemblance to twofactor theory have been proposed (Grossberg, 1972;Johnson, Li, Li, & Klopf, 2002;Schmajuk & Zanutto, 1997). Of these, the model with the closest resemblances to the Two-factor theory, the actor-critic model, and conditioned avoidance…”