Reinforcement learning with artificial microswimmers

Abstract

Artificial microswimmers that can replicate the complex behavior of active matter are often designed to mimic the self-propulsion of microscopic living organisms. However, compared with their living counterparts, artificial microswimmers have a limited ability to adapt to environmental signals or to retain a physical memory to yield optimized emergent behavior. Different from macroscopic living systems and robots, both microscopic living organisms and artificial microswimmers are subject to Brownian motion, which randomizes their position and propulsion direction. Here, we combine real-world artificial active particles with machine learning algorithms to explore their adaptive behavior in a noisy environment with reinforcement learning. We use a real-time control of self-thermophoretic active particles to demonstrate the solution of a simple standard navigation problem under the inevitable influence of Brownian motion at these length scales. We show that, with external control, collective learning is possible. Concerning the learning under noise, we find that noise decreases the learning speed, modifies the optimal behavior, and also increases the strength of the decisions made. As a consequence of time delay in the feedback loop controlling the particles, an optimum velocity, reminiscent of optimal run-and-tumble times of bacteria, is found for the system, which is conjectured to be a universal property of systems exhibiting delayed response in a noisy environment.

Navigation of micro-swimmers in steady flow: the importance of symmetries

“…With Q-learning, we search for optimal strategies for fast vertical migration in a two-dimensional steady Taylor-Green vortex flow (Taylor 1923), which allows for direct comparison with Colabrese et al (2017), and in a steady two-dimensional random velocity field. We investigate how symmetry breaking allows the swimmer to distinguish different directions (which is necessary to swim upwards), which highlights significant differences between navigation using local signals in the frame of reference of the swimmer and signals in the laboratory frame (Colabrese et al 2017(Colabrese et al , 2018Gustavsson et al 2017;Biferale et al 2019;Schneider & Stark 2019;Alageshan et al 2020;Gunnarson et al 2021;Muiños-Landin et al 2021). We find that settling owing to gravity allows the swimmers to find efficient strategies for vertical migration, because the settling breaks vertical reflection symmetry.…”

“…In all of these studies, some signals and actions referred to the laboratory frame, so that the swimmer had, in effect, access to a map, which facilitated navigation. The navigation problems considered in recent studies (Biferale et al 2019;Schneider & Stark 2019;Gunnarson et al 2021;Muiños-Landin et al 2021) also used information relating to a fixed reference frame.…”

Microswimmers learning chemotaxis with genetic algorithms

“…Notably, two contributions have taken the viscous environment into account, namely one applying Q-learning to a three-bead-swimmer (29) and one using deep learning to find energetically efficient collective swimming of fish (30). Experimental realizations of ML applied to self-propelled objects are navigation of microswimmers on a grid (31) or macroscopic gliders learning to soar in the atmosphere (32).…”

Control and controllability of microswimmers by a shearing flow

“…As such, careful evaluation of diffusive effects is warranted in general, with the potential for complex interplay between multiple aspects of the control system. Though we have only considered prescribed controls in this work, addressing the impacts of noise may also warrant the use of control-feedback loops, in which the control is adjusted in real time to account for the measured impacts of noise or other complicating factors [ 6 , 55 ].…”