Identification of an unconventional process of instrumental learning characteristically initiated with outcome devaluation-insensitivity and generalized action selection

Abstract

The distinction between goal-directed action and habitual response, particularly with respect to moderate or extended appetitive instrumental training, is well documented; however, the propensity toward instrumental behavior in the early training stage has not been elucidated. In this study, we trained Sprague Dawley rats to press a lever to obtain food as an outcome for various time periods and monitored the changes in their sensitivity to outcome devaluation and choice between the levers they had been trained with and unfamiliar levers. After the extensive training with a random interval schedule, the rats were insensitive to outcome devaluation, and exhibited a typical habit-like phenotype, as previously reported, and the untrained leverpresses were relatively rare and sporadic. During the initial stage of training (≤1 week), the rats exhibited a similar insensitivity to the devaluation; however, in contrast to the overtrained condition, they performed distinctive unbiased leverpresses on both the trained and untrained levers. Thus, we propose a possibility that, contrary to the authentic concept that instrumental learning is initiated with an outcome devaluation-sensitive goal-directed stage, under some conditions, this learning can unconventionally begin with the initial stage that is distinct from both goal-directed action and habitual response.

Lever Insertion as a Salient Stimulus Promoting Insensitivity to Outcome Devaluation

“…Although automaticity and low uncertainty correlates with expression of habit in the DT5 procedure, it is important to note that the FR5 and DT5 experiments were run separately, which precludes strong conclusions about causality between these factors. Furthermore, instrumental performance can be both automatic (i.e., efficiency, discrimination, action chunking) and goal-directed (Derusso et al, 2010 ; Iguchi et al, 2017 ), suggesting that stronger automaticity is the DT5 procedure is not sufficient to explain the rapid development of habit observed in this study. Finally, although differences in automaticity of behavior are consistent with the opposite response strategies observed in the DT5 and FR5 procedures when rats are trained with sucrose, it cannot explain why rats trained with grain-based pellets in the DT5 procedure remained sensitive to satiety-induced devaluation (discussed above).…”

Orbital Frontal Cortex Projections to Secondary Motor Cortex Mediate Exploitation of Learned Rules

“…We next sought to determine if the amount of experience with the learned action biased towards exploitation, as previously reported 30 . Utilizing data obtained from the mice in the uncertainty experiment above, we calculated the total lever presses made since the start of schedule training until either the early or late generalization test.…”

“…If novel lever pressing is not the sole result of generalization of learned rules, or of stimulus-response associations, what is controlling responding? It has recently been proposed that exploration is a distinct, early stage of learning which disappears following extended training 30 . If exploration disappeared with training, we should expect a negative correlation between the amount of experience an animal had with the instrumental contingency and novel lever pressing.…”

“…Different schedules of reinforcement can be used to bias either exploitation of the trained lever or exploration of the novel lever 28 , 29 . Previous studies using this particular task have hypothesized that responding reflects either exploration 28 , 30 or action generalization mechanisms 29 , though this has not been tested.…”

“…We used a self-paced operant instrumental lever press task in mice to determine if exploration utilizes learned rules and the extent to which exploration and exploitation directly compete. In this task 28 – 30 , mice are trained to press one lever for a food reward. Then during the test session a novel but perceptually similar lever is also inserted into the chamber, and we measure responses on the trained and novel levers.…”