The actions of others act as a pseudo-reward to drive imitation in the context of social reinforcement learning

Abstract

While there is no doubt that social signals affect human reinforcement learning, there is still no consensus about how this process is computationally implemented. To address this issue, we compared three psychologically plausible hypotheses about the algorithmic implementation of imitation in reinforcement learning. The first hypothesis, decision biasing (DB), postulates that imitation consists in transiently biasing the learner’s action selection without affecting their value function. According to the second hypothesis, model-based imitation (MB), the learner infers the demonstrator’s value function through inverse reinforcement learning and uses it to bias action selection. Finally, according to the third hypothesis, value shaping (VS), the demonstrator’s actions directly affect the learner’s value function. We tested these three hypotheses in 2 experiments (N = 24 and N = 44) featuring a new variant of a social reinforcement learning task. We show through model comparison and model simulation that VS provides the best explanation of learner’s behavior. Results replicated in a third independent experiment featuring a larger cohort and a different design (N = 302). In our experiments, we also manipulated the quality of the demonstrators’ choices and found that learners were able to adapt their imitation rate, so that only skilled demonstrators were imitated. We proposed and tested an efficient meta-learning process to account for this effect, where imitation is regulated by the agreement between the learner and the demonstrator. In sum, our findings provide new insights and perspectives on the computational mechanisms underlying adaptive imitation in human reinforcement learning.

Conformist social learning leads to self-organised prevention against adverse bias in risky decision making

“…We obtained similar results across different settings of the multi-armed bandit tasks such as skewed payoff distribution from which large payoffs realised less likely whereas small payoffs realised commonly due to an asymmetric probability unlike the Gaussian noise (March, 1996; Denrell, 2007) (Supporting Figure S11). Further, the conclusion still held under different assumptions of social influences on reinforcement learning, assuming that the conformity-biased influence acts on the learning process (the value-shaping model (Najar et al, 2020)) rather than on the action-selection (the decision-biasing model) assumed above (Supporting Methods). Although groups of agents with the value-shaping algorithm seemed more prone to herding, the results were not qualitatively changed and collective behavioural rescue emerged when β is sufficiently small (Supporting Figure S9).…”

“…We considered another implementation of social influences in the reinforcement learning, namely, the value-shaping process (Biele et al, 2011; Najar et al, 2020) rather than the decision-biasing process assumed in our main analyses. In the value-shaping model, social influence modifies the Q value’s updating process as follows: where the social frequency cue acts as an additional ‘bonus’ of the value that was weighted by σ vs ( σ vs > 0) and standardised by the expected payoff from choosing randomly among all alternatives π .…”

“…We considered another implementation of social influences in the reinforcement learning, namely, the value-shaping process (Biele et al, 2011;Najar et al, 2020) rather than the decision-biasing process assumed in our main analyses.…”

“…10). Although it is out of the focus of this article, whether people's behaviour can be fit better by this value-shaping model than by the decision-biasing model is an interesting question for the future research (Najar et al, 2020).…”

“…S6). Further, the conclusion still held under different assumptions of social influences on reinforcement learning, assuming that the conformist influence acts on the learning process (the value-shaping model (Najar et al, 2020)) rather than on the action-selection process (the decisionbiasing model) assumed above (see Supplementary Methods). Although groups of agents with the value-shaping algorithm seemed more prone to herding, the results were not qualitatively changed and collective behavioural rescue emerged when was sufficiently small (Supporting Fig.…”