Synergistic effects of adaptive reward and reinforcement learning rules on cooperation

Abstract

Understanding cooperative behavior in multi-agent system is a research hotspot. In the context of pairwise interaction games, several researches have used reinforcement learning rules to successfully explain and predict the behavior of agents. However, multi-agent interactions are more general than two-agent interactions, and the effect of reward mechanism on behavior of agents is also ignored under the reinforcement learning rules. Therefore, this paper established a framework that combines the public goods game with reinforcement learning and adaptive reward. In that, public goods game is adopted to reflect the decision-making behavior of multi-agent interactions, self-regarding Q-learning emphasizes an experience-based strategy update, and adaptive reward focuses on the adaptability. We are mainly concentrating on the synergistic effects of them. It is remarkable that while self-regarding Q-learning fails to prevent the collapse of cooperation in the traditional public goods game, the fraction of cooperation increases significantly when the adaptive reward strategy is included. Meanwhile, the theoretical analysis results match well with the simulation results, which indicate that there is a specific reward cost to maximize the fraction of cooperation. Our findings may provide a new perspective for establishing cooperative reward mechanisms in social dilemmas.

The emergence of cooperation via Q-learning in spatial donation game

“…Moreover, in contrast to existing works that combine RL with game theory [55,59,70], our model incorporate more environmental information (neighbors' decisions) to expand the state space, which enables individuals to more effectively tailor their responses to diverse environments, ultimately enhancing the overall performance of the population. It may provide a relatively simple explanation for the emergence of cooperation under this introspective learning mechanism.…”

“…Zhang et al [56,57] explored the emergence of cooperation characterized by explosiveness and cyclic oscillations in two-player games, and found that the herding effect could be effectively eliminated in minority games [58]. Furthermore, the incorporation of other mechanisms, such as adaptive reward [59], social payoff [55], extortion strategy [60], and Lévy noise [61], has shown to significantly improve the level of cooperation. Nonetheless, the self-regarding Q-learning approach, which emphasizes policy updates based on one's experience, leads to an alignment of individual state space with strategy space, thereby overlooking environmental cues.…”