Oscillatory evolution of collective behavior in evolutionary games played with reinforcement learning

Zhang, Siping; Zhang, Jiqiang; Chen, Li; Li, Xudong

doi:10.1007/s11071-019-05398-4

Cited by 37 publications

(12 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is the mainstream paradigm of modeling updating. However, a new paradigm proposed recently [54,55] is through the inward learning, where the decision-making of individuals is through introspective actions based on their history. With the help of machine learning [56], they also model the evolution of cooperation, but is limited to the single game case.…”

Section: Summary and Discussionmentioning

confidence: 99%

Dynamical reciprocity in interacting games: numerical results and mechanism analysis

Liang,

Wang,

Zhang

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We study the evolution two mutually interacting games with both pairwise games as well as the public goods game on different topologies. On 2d square lattices, we reveal that the game-game interaction can promote the cooperation prevalence in all cases, and the cooperation-defection phase transitions even become absent and fairly high cooperation is expected when the interaction goes to be extremely strong. A mean-field theory is developed that points out new dynamical routes arising therein. Detailed analysis shows indeed that there are rich categories of interactions in either individual or bulk scenario: invasion, neutral, and catalyzed types; their combination put cooperators at a persistent advantage position, which boosts the cooperation. The robustness of the revealed reciprocity is strengthened by the studies of model variants, including asymmetrical or time-varying interactions, games of different types, games with time-scale separation, different updating rules etc. The structural complexities of the underlying population, such as Newman-Watts small world networks, Erdős-Rényi random networks, and Barabási-Albert networks, also do not alter the working of the dynamical reciprocity. In particular, as the number of games engaged increases, the cooperation level continuously improves in general. Our work thus uncovers a new class of cooperation mechanism and indicates the great potential for human cooperation where concurrent issues are so often seen in the real world.

show abstract

Section: Summary and Discussionmentioning

confidence: 99%

Dynamical reciprocity in interacting games: numerical results and mechanism analysis

Liang,

Wang,

Zhang

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Q-values are defined by Q-table to record the relative utility of different actions in different states. As per Zhang and co-workers [41,42], the state set bold-italicS and action set bold-italicA are supposed to be the same, i.e. bold-italicS=bold-italicA=falsefalse{C,Dfalsefalse}.…”

Section: Methodsmentioning

confidence: 99%

“…Then, the agent with the selected action interacts with their neighbours and calculates payoffs. After that, the Q-value is updated by the following equation[41,43]: Qs,afalse(t+1false)=false(1−ηfalse)Qs,afalse(tfalse)+ηfalse[Πfalse(tfalse)+γQs′,a′max(t)false],where η∈false(0,1false] and γ∈false[0,1false) denote the learning rate and discount factor (foresight level of agents), respectively. Πfalse(tfalse) is the calculated payoff.…”

Section: Methodsmentioning

confidence: 99%

“…This self-regarding decision-making process can be well described by reinforcement learning [38–40]. Q-learning is a typical reinforcement learning approach that has been demonstrated to be a promising method for investigating the emergence of cooperative behaviour in agents [41–45]. As a strategy update rule, a table (each item corresponds to a combination of a state and an action) is used to store the Q-values, and the optimal action is generated by executing the action with maximum Q-value in the current state.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Extending Q -learning to continuous and mixed strategy games based on spatial reciprocity

et al. 2023

View full text Add to dashboard Cite

The discrete strategy game, in which agents can only choose cooperation or defection, has received lots of attention. However, this hypothesis seems implausible in the real world, where choices may be continuous or mixed. Furthermore, when applying Q -learning to continuous or mixed strategy games, one of the challenges is that the learning space grows drastically as the number of states and actions rises. So, in this article, we redesign the Q -learning method by considering the spatial reciprocity, in which agents simply interact with their four neighbours to get the reward and learn the action by taking neighbours’ strategy into account. As a result, the learning state and action space is transformed into a 5 × 5 table that stores the state and action of the focal agent and its four neighbours, avoiding the curse of dimensionality caused by a continuous or mixed strategy game. The numerical simulation results reveal the striking differences between the three classes of games. In detail, the discrete strategy game is more vulnerable to the setting of relevant parameters, whereas the other two strategy games are relatively stable. At the same time, in terms of promoting cooperation, a mixed strategy game is always better than a continuous one.

show abstract

“…Recently, Yang et al [32] observed a tide-like burst in a probabilisitic migration model of prisoner's dilemma, where migration is driven by conformity and self-centered inequity aversion norms. It's also worth noting that recently Zhang et al [33] introduced the reinforcement learning framework, where periodic burst-like oscillation is also seen in pairwise games.…”

Section: Introductionmentioning

confidence: 99%

Oscillatory cooperation prevalence emerges from misperception

Zhang¹,

Zhao²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Oscillatory behaviors are ubiquitous in nature and the human society. However, most previous works fail to reproduce them in the two-strategy game-theoretical models.Here we show that oscillatory behaviors naturally emerge if incomplete information is incorporated into the cooperation evolution of a non-Markov model. Specifically, we consider a population playing prisoner's dilemma game, where each individual can only probabilistically get access to their neighbors' payoff information and store them within their memory with a given length. They make their decisions based upon these memories. Interestingly, we find that the level of cooperation generally cannot stabilize but render quasiperiodic oscillation, and this observation is strengthened for a longer memory and a smaller information acquisition probability. The mechanism uncovered shows that there are misperceived payoffs about the player's neighborhood, facilitating the growth of cooperators and defectors at different stages that leads to oscillatory behaviors as a result. Our findings are robust to the underlying structure of the population. Given the omnipresence of incomplete information, our findings may provide a plausible explanation for the phenomenon of oscillatory behaviors in the real world.

show abstract

Oscillatory evolution of collective behavior in evolutionary games played with reinforcement learning

Cited by 37 publications

References 65 publications

Dynamical reciprocity in interacting games: numerical results and mechanism analysis

Dynamical reciprocity in interacting games: numerical results and mechanism analysis

Extending Q -learning to continuous and mixed strategy games based on spatial reciprocity

Oscillatory cooperation prevalence emerges from misperception

Contact Info

Product

Resources

About