Smart Magnetic Microrobots Learn to Swim with Deep Reinforcement Learning

Abstract

Swimming microrobots are increasingly developed with complex materials and dynamic shapes and are expected to operate in complex environments in which the system dynamics are difficult to model and positional control of the microrobot is not straightforward to achieve. Deep reinforcement learning is a promising method of autonomously developing robust controllers for creating smart microrobots, which can adapt their behavior to operate in uncharacterized environments without the need to model the system dynamics. This article reports the development of a smart helical magnetic hydrogel microrobot that uses the soft actor critic reinforcement learning algorithm to autonomously derive a control policy which allows the microrobot to swim through an uncharacterized biomimetic fluidic environment under control of a time‐varying magnetic field generated from a three‐axis array of electromagnets. The reinforcement learning agent learns successful control policies from both state vector input and raw images, and the control policies learned by the agent recapitulate the behavior of rationally designed controllers based on physical models of helical swimming microrobots. Deep reinforcement learning applied to microrobot control is likely to significantly expand the capabilities of the next generation of microrobots.

A Survey on Reinforcement Learning Methods in Bionic Underwater Robots

“…In addition, most bionic underwater robots have periodic motion, so real-time output of the reinforcement learning control algorithm is not necessary, and periodic control output is more suitable for the needs of bionic underwater robots. The learned policy’s action distribution via regression is fitted as mathematical functions in [ 134 ], so that the reinforcement learning control strategy can be fine-tuned after algorithm deployment. Moreover, centralized training with decentralized execution (CTDE) is a common training paradigm for swarm tasks [ 16 ].…”

AI-enhanced biomedical micro/nanorobots in microfluidics

“…114 For gait optimization, recent studies using RL and genetic algorithms for swimming strategy optimization in different microswimmers have been explored. [115][116][117] Hartl et al employed the NEAT (NeuroEvolution of Augmenting Topologies) genetic algorithm and artificial neural networks to control the motion of microswimmers. This method obtained inspiration from biologically relevant chemotactic sensing strategies.…”

Hard magnetics and soft materials—a synergy

“…Recently, Yao et al [356] have used RL algorithms for an intelligent approach that tackled the inverse problem of discovering feasible magnetic fields for actuating strip-like soft robots. By employing a helical magnetic hydrogel microrobot, controlled through RL algorithms, the study by Behrens and Ruder [361] demonstrated the capability of the robot to autonomously navigate complex environments. The soft robot successfully learned control strategies from various inputs and replicated behaviors seen in conventional controllers based on physical models.…”

“…These models, inspired by the concept of trial and error, enable systems to make decisions based on past experiences and rewards [360]. In hMSM research, RL approaches facilitate the exploration of optimal material configurations and their potential applications in various industries [361,362].…”