From Simulation to Reality: A Learning Framework for Fish-Like Robots to Perform Control Tasks

Zhang, Tianhao; Tian, Runyu; Yang, Hongqi; Wang, Chen; Sun, Jing; Zhang, Shikun; Xie, Guangming

doi:10.1109/tro.2022.3181014

Cited by 20 publications

(12 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the field of motion control for robotic fish, Xie’s team has worked on a DRL-based control method [ 17 , 18 , 98 , 114 ], demonstrating a complete and informative RL motion control framework. To address an unknown flow field, the study by [ 17 ] fuses data from LLS and IMU, and trains a data-driven simulation environment based on DDPG, holding a desired angle of attack.…”

Section: Rl-based Methods In Task Spaces Of Bionic Underwater Robotsmentioning

confidence: 99%

“…To address an unknown flow field, the study by [ 17 ] fuses data from LLS and IMU, and trains a data-driven simulation environment based on DDPG, holding a desired angle of attack. In addition, the studies [ 18 , 98 ] focus on path tracking and pose control, and train in both the surrogate environment and the CFD environment based on A2C, improving the efficiency of RL training and the precision of underwater control experiments. To balance position control and attitude control, ref.…”

Section: Rl-based Methods In Task Spaces Of Bionic Underwater Robotsmentioning

confidence: 99%

“…The computational complexity during training is related to the specific RL method chosen, the training environment, and the algorithm convergence speed, which usually involves complex calculations and takes a long time. On the other hand, the computational complexity during deployment for some RL algorithms often only requires a few matrix operations to be completed [ 18 ]. Bionic underwater robot platforms usually have a small hull and limited computing power.…”

Section: Training and Deployment Methods Of Rl On Bionic Underwater R...mentioning

confidence: 99%

“…In addition, ref. [ 18 ] pointed out the training period of the proposed algorithm. The whole learning process takes 16 days, where the first 350 episodes within the surrogate environment take only 50 min [ 18 ].…”

Section: Training and Deployment Methods Of Rl On Bionic Underwater R...mentioning

confidence: 99%

“…Second, the underwater environment is much more unstable, and not only is it susceptible to interference with the swing of the robot’s body and fin, it is also not easily modeled [ 17 ]. Under the disturbances of the aquatic environment and the influence of strong nonlinearities, achieving satisfactory control performance for bionic robots remains challenging [ 18 ].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

A Survey on Reinforcement Learning Methods in Bionic Underwater Robots

Feng

Wang

et al. 2023

Biomimetics

View full text Add to dashboard Cite

Bionic robots possess inherent advantages for underwater operations, and research on motion control and intelligent decision making has expanded their application scope. In recent years, the application of reinforcement learning algorithms in the field of bionic underwater robots has gained considerable attention, and continues to grow. In this paper, we present a comprehensive survey of the accomplishments of reinforcement learning algorithms in the field of bionic underwater robots. Firstly, we classify existing reinforcement learning methods and introduce control tasks and decision making tasks based on the composition of bionic underwater robots. We further discuss the advantages and challenges of reinforcement learning for bionic robots in underwater environments. Secondly, we review the establishment of existing reinforcement learning algorithms for bionic underwater robots from different task perspectives. Thirdly, we explore the existing training and deployment solutions of reinforcement learning algorithms for bionic underwater robots, focusing on the challenges posed by complex underwater environments and underactuated bionic robots. Finally, the limitations and future development directions of reinforcement learning in the field of bionic underwater robots are discussed. This survey provides a foundation for exploring reinforcement learning control and decision making methods for bionic underwater robots, and provides insights for future research.

show abstract

Section: Rl-based Methods In Task Spaces Of Bionic Underwater Robotsmentioning

confidence: 99%

Section: Rl-based Methods In Task Spaces Of Bionic Underwater Robotsmentioning

confidence: 99%

Section: Training and Deployment Methods Of Rl On Bionic Underwater R...mentioning

confidence: 99%

Section: Training and Deployment Methods Of Rl On Bionic Underwater R...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Survey on Reinforcement Learning Methods in Bionic Underwater Robots

Feng

Wang

et al. 2023

Biomimetics

View full text Add to dashboard Cite

show abstract

Learn to flap: foil non-parametric path planning via deep reinforcement learning

Wang,

Lin,

Zhao

et al. 2024

J. Fluid Mech.

View full text Add to dashboard Cite

To optimize flapping foil performance, in the current study we apply deep reinforcement learning (DRL) to plan foil non-parametric motion, as the traditional control techniques and simplified motions cannot fully model nonlinear, unsteady and high-dimensional foil–vortex interactions. Therefore, a DRL training framework is proposed based on the proximal policy optimization algorithm and the transformer architecture, where the policy is initialized from the sinusoidal expert display. We first demonstrate the effectiveness of the proposed DRL-training framework, learning the coherent foil flapping motion to generate thrust. Furthermore, by adjusting reward functions and action thresholds, DRL-optimized foil trajectories can gain significant enhancement in both thrust and efficiency compared with the sinusoidal motion. Last, through visualization of wake morphology and instantaneous pressure distributions, it is found that DRL-optimized foil can adaptively adjust the phases between motion and shedding vortices to improve hydrodynamic performance. Our results give a hint of how to solve complex fluid manipulation problems using the DRL method.

show abstract

A numerical simulation research on fish adaption behavior based on deep reinforcement learning and fluid–structure coupling: Implementation of the “perceive-feedback-memory” control system

Zhang,

Li,

Zhang

et al. 2024

Physics of Fluids

View full text Add to dashboard Cite

The autonomous swimming of fish in a complex flow environment is a nonlinear and intricate system, which is the focus and challenge in various fields. This study proposed a novel simulation framework for artificial intelligence fish. It employed a high-precision immersed boundary-lattice Boltzmann coupling scheme to simulate the interactions between fish and flow in real time, and utilized the soft actor-critic (SAC) deep reinforcement learning algorithm for fish brain decision-making module, which was further divided into a vision-based directional navigation and a lateral line-based flow perception modules, each matched with its corresponding macro-action space. The flow features were extracted using a deep neural network based on a multi-classification algorithm from the data perceived by the lateral line and were linked to the fish actions. The predation swimming and the various Kármán gait swimming were explored in terms of training, simulation, and generalization. Numerical results demonstrated significant advantages in the convergence speed and training efficiency of the SAC algorithm. Owing to the closed-loop “perceive-feedback-memory” mode, intelligent fish can respond in real-time to changes in flow fields based on reward-driven requirements and experience, and the accumulated experience can be directly utilized in other flow fields, and its adaptability, model training efficiency, and generalization were substantially improved.

show abstract

From Simulation to Reality: A Learning Framework for Fish-Like Robots to Perform Control Tasks

Cited by 20 publications

References 35 publications

A Survey on Reinforcement Learning Methods in Bionic Underwater Robots

A Survey on Reinforcement Learning Methods in Bionic Underwater Robots

Learn to flap: foil non-parametric path planning via deep reinforcement learning

A numerical simulation research on fish adaption behavior based on deep reinforcement learning and fluid–structure coupling: Implementation of the “perceive-feedback-memory” control system

Contact Info

Product

Resources

About