A-learning: A new formulation of associative learning theory

Abstract

We present a new mathematical formulation of associative learning focused on non-human animals, which we call Alearning. Building on current animal learning theory and machine learning, A-learning is composed of two learning equations, one for stimulus-response values and one for stimulus values (conditioned reinforcement). A third equation implements decision-making by mapping stimulus-response values to response probabilities. We show that A-learning can reproduce the main features of: instrumental acquisition, including the effects of signaled and unsignaled non-contingent reinforcement; Pavlovian acquisition, including higher-order conditioning, omission training, autoshaping, and differences in form between conditioned and unconditioned responses; acquisition of avoidance responses; acquisition and extinction of instrumental chains and Pavlovian higher-order conditioning; Pavlovian-to-instrumental transfer; Pavlovian and instrumental outcome revaluation effects, including insight into why these effects vary greatly with training procedures and with the proximity of a response to the reinforcer. We discuss the differences between current theory and A-learning, such as its lack of stimulus-stimulus and response-stimulus associations, and compare A-learning with other temporal-difference models from machine learning, such as Q-learning, SARSA, and the actor-critic model. We conclude that A-learning may offer a more convenient view of associative learning than current mathematical models, and point out areas that need further development.

Benefitting from trial spacing without the cost of prolonged training: Frequency, not duration, of trials with absent stimuli enhances perceived contingency

“…Second, there are trial-based models (e.g., Rescorla & Wagner, 1972) which do not explicitly code the duration of events; this appears problematic given the often reported relevance of temporal information as a learning factor in both humans and animals (e.g., Buriticá & Alcalá, 2019;Nasser & Delamater, 2016). However, a number of attempts to integrate timing into the classic trial-based models have been suggested (e.g., Delamater, Desouza, Rivkin, & Derman, 2014;Donahoe, Burgos, & Palmer, 1993;Gershman, 2015;Ghirlanda, Lind, & Enquist, 2020;Luzardo, Alonso, & Mondragon, 2017). Third, there are behavioural timing models (e.g., Killeen & Fetterman, 1988;Machado, 1997), in which agent's internal states (behavioural states) serve as events that are associated with moments in the environment at a particular time.…”

Can associative learning be the general process for intelligent behavior in non-human animals?

“…Besides, it can account for general observations of social learning in animals [35] and planning behavior observed in great apes and ravens [26]. In addition, this model can reproduce core features of results from animal psychology, for example instrumental and Pavlovian acquisition, conditioned reinforcement, and different kinds of higher-order conditioning [30]. The general nature of the included benchmark tests can further our knowledge of what role associative learning can play for animal intelligence.…”

“…The winner used a common three-step approach where first training environments were made, after which an agent received training, ending with a validation step involving behavior analysis. These Reinforcement Learning algorithms and recent models in animal learning research have much in common [28][29][30], and by taking both general process and adaptive specialization into account they may provide useful tools for working towards a synthesis of animal intelligence. By bridging the gap between comparative cognition and AI, The Animal-AI Olympics provide a new benchmark selection of tests different from other AI tests investigating intelligence, such as Arcade Learning Environment (ALE) [31], OpenAI Gym [32], and General Video Game AI [33].…”

“…In this paper, we take advantage of the benchmark tests selected for The Animal-AI Olympics [23,24] and subject a new associative learning model -A-learning -to these tests [30,34]. This recently developed model can produce long behavioral sequences and can yield optimal solutions to problems encountered by an agent.…”

“…To subject this new model to the different tests, we translated the tasks included in The Animal-AI Olympics into script-based description of the environments and their reward structures. As this model does not include perceptual and motor mechanisms, environments and agents' behaviors are translated into verbal descriptions of key stimuli and actions, as in previous computational analyses of learning phenomena with this model [26,30,34,35]. In computer simulations, the agent learns from actions it performs to navigate the world of stimuli and rewards.…”