Learning to flock through reinforcement

Abstract

Flocks of birds, schools of fish, insects swarms are examples of coordinated motion of a group that arises spontaneously from the action of many individuals. Here, we study flocking behavior from the viewpoint of multi-agent reinforcement learning. In this setting, a learning agent tries to keep contact with the group using as sensory input the velocity of its neighbors. This goal is pursued by each learning individual by exerting a limited control on its own direction of motion. By means of standard reinforcement learning algorithms we show that: i) a learning agent exposed to a group of teachers, i.e. hard-wired flocking agents, learns to follow them, and ii) that in the absence of teachers, a group of independently learning agents evolves towards a state where each agent knows how to flock. In both scenarios, i) and ii), the emergent policy (or navigation strategy) corresponds to the polar velocity alignment mechanism of the well-known Vicsek model. These results show that a) such a velocity alignment may have naturally evolved as an adaptive behavior that aims at minimizing the rate of neighbor loss, and b) prove that this alignment does not only favor (local) polar order, but it corresponds to best policy/strategy to keep group cohesion when the sensory input is limited to the velocity of neighboring agents. In short, to stay together, steer together.

Development of swarm behavior in artificial learning agents that adapt to different foraging environments

“…[53,63]) consider a variant of the above distribution that only follows the PL form for steps longer than some threshold ℓ 0 , for example when analysing experimental data that become increasingly noisy at short step-lengths. However, since the step lengths resulting from our simulations are natively discrete, the unbounded PL distribution given in Eq (14) seems appropriate. Moreover, if one were to introduce a lower bound ℓ 0 > 1, one would need to add more parameters in the model to account for the probabilities p(ℓ) for all 1 � ℓ < ℓ 0 , which we consider an unnecessary complication.…”

“…This type of agent-based models that employ artificial intelligence to model behavior are gaining popularity in the last few years. Artificial neural networks (ANN) have been used, for instance, in the context of navigation behaviors [10,11] and reinforcement learning (RL) algorithms have been applied to model collective behavior in different scenarios, such as pedestrian movement [12] or flocking [13,14].…”

“…Since the interaction rules are developed by the agents themselves, the challenge is to design the environment and learning task that will give rise to the individual and, consequently, collective behavior. In previous works, the agents are directly rewarded for aligning themselves with the surrounding agents [15] or for not losing neighbours [14]. Instead of rewarding a specific behavior, in this work we set a survival task that the agents need to fulfill in order to get the reward, and then analyze the emergent behavioral dynamics.…”

A least microenvironmental uncertainty principle (LEUP) as a generative model of collective cell migration mechanisms

“…As stated above, LEUP circumvents the biophysical details of cell migration. The need to model systems of interacting agents without previous knowledge of the biophysical mechanisms involved has sparked at least another agent based model 41 . In this model, similarly to ours, agents act without a mechanistic rule.…”

“…Rather, they consider every possible action and penalize those which are not favorable to their internal standards. While both the aforementioned model and LEUP are defined in a similar spirit, modeling under LEUP consists in correctly identifying the relevant internal cellular states for entropy optimization, while in 41 modeling is concerned with defining suitable penalizations for each possible decision scenario.…”