Reinforcement-learning-based control of convectively unstable flows

Xu, Da; Zhang, Mengqi

doi:10.1017/jfm.2022.1020

Cited by 7 publications

(6 citation statements)

References 101 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Paris, Beneddine & Dandois (2023) proposed an RL methodology to optimise actuator placement in a laminar 2-D flow around an aerofoil, addressing the trade-off between performance and the number of actuators. Xu & Zhang (2023) used RL to suppress instabilities in both the Kuramoto-Sivashinsky system and 2-D boundary layers, showing the effectiveness and robustness of RL control. Pino et al (2023) compared RL and genetic programming algorithms to global optimisation techniques for various cases, including the viscous Burger's equation and vortex shedding behind a 2-D cylinder.…”

Section: Model-free Active Flow Control By Reinforcement Learningmentioning

confidence: 99%

“…Sonoda et al (2023) and Guastoni et al (2023) applied RL control in numerical simulations of turbulent channel flow, and showed that RL control can outperform opposition control in this complex flow control task. Some RL techniques have been applied also to various flow control problems with different geometries, such as flow past a 2-D cylinder , vortex-induced vibration of a 2-D square bluff body (Chen et al 2023), and a 2-D boundary layer (Xu & Zhang 2023). However, model-free RL control techniques also have several drawbacks compared to model-based control.…”

Section: Model-free Active Flow Control By Reinforcement Learningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Active flow control for bluff body drag reduction using reinforcement learning with partial measurements

Xia,

Zhang,

Kerrigan

et al. 2024

J. Fluid Mech.

View full text Add to dashboard Cite

Active flow control for drag reduction with reinforcement learning (RL) is performed in the wake of a two-dimensional square bluff body at laminar regimes with vortex shedding. Controllers parametrised by neural networks are trained to drive two blowing and suction jets that manipulate the unsteady flow. The RL with full observability (sensors in the wake) discovers successfully a control policy that reduces the drag by suppressing the vortex shedding in the wake. However, a non-negligible performance degradation ( $\sim$ 50 % less drag reduction) is observed when the controller is trained with partial measurements (sensors on the body). To mitigate this effect, we propose an energy-efficient, dynamic, maximum entropy RL control scheme. First, an energy-efficiency-based reward function is proposed to optimise the energy consumption of the controller while maximising drag reduction. Second, the controller is trained with an augmented state consisting of both current and past measurements and actions, which can be formulated as a nonlinear autoregressive exogenous model, to alleviate the partial observability problem. Third, maximum entropy RL algorithms (soft actor critic and truncated quantile critics) that promote exploration and exploitation in a sample-efficient way are used, and discover near-optimal policies in the challenging case of partial measurements. Stabilisation of the vortex shedding is achieved in the near wake using only surface pressure measurements on the rear of the body, resulting in drag reduction similar to that in the case with wake sensors. The proposed approach opens new avenues for dynamic flow control using partial measurements for realistic configurations.

show abstract

Section: Model-free Active Flow Control By Reinforcement Learningmentioning

confidence: 99%

Section: Model-free Active Flow Control By Reinforcement Learningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Active flow control for bluff body drag reduction using reinforcement learning with partial measurements

Xia,

Zhang,

Kerrigan

et al. 2024

J. Fluid Mech.

View full text Add to dashboard Cite

show abstract

“…166 and 167) body around a larger one with the prospect of reducing the recirculation bubble/the drag downstream, to directly modify the shape of the principal body 52,[168][169][170] or to control its movements 102,153 with the same objective of optimizing its aerodynamic/hydrodynamic properties. Other research works handle heat transport Deep reinforcement learning for flow control: perspectives and future directions issues such as the Rayleigh-Bénard instability 127 , or other convectively-unstable flows 171 .…”

Section: Deep Reinforcement Learning and Active Flow Control: Challen...mentioning

confidence: 99%

Recent advances in applying deep reinforcement learning for flow control: Perspectives and future directions

2023

View full text Add to dashboard Cite

Deep reinforcement learning (DRL) has been applied to a variety of problems during the past decade, and has provided effective control strategies in high-dimensional and non-linear situations that are challenging to traditional methods. Flourishing applications now spread out into the field of fluid dynamics, and specifically of active flow control (AFC). In the community of AFC, the encouraging results obtained in two-dimensional and chaotic conditions have raised interest to study increasingly complex flows. In this review, we first provide a general overview of the reinforcement-learning (RL) and DRL frameworks, as well as their recent advances. We then focus on the application of DRL to AFC, highlighting the current limitations of the DRL algorithms in this field, and suggesting some of the potential upcoming milestones to reach, as well as open questions that are likely to attract the attention of the fluid-mechanics community.

show abstract

“…An other example is the work of Gunnarson et al (2021) who explored the influence of the input data provided to the RL algorithm to observe the state of the environment, and compared these with optimal control, aware of the full flow field. Other studies of input data optimized the sensor layout feeding the algorithm (Paris et al, 2021; Xu and Zhang, 2023), or, in an experimental context, performed high-frequency filtering of the state to enable the agent to learn a successful policy (Fan et al, 2020). Another element of tremendous importance is the feedback signal provided to the RL algorithm indicating how well it performed.…”

Section: Introductionmentioning

confidence: 99%

Reliability assessment of off-policy deep reinforcement learning: A benchmark for aerodynamics

Berger,

Arroyo Ramo,

Guillet

et al. 2024

DCE

View full text Add to dashboard Cite

Deep reinforcement learning (DRL) is promising for solving control problems in fluid mechanics, but it is a new field with many open questions. Possibilities are numerous and guidelines are rare concerning the choice of algorithms or best formulations for a given problem. Besides, DRL algorithms learn a control policy by collecting samples from an environment, which may be very costly when used with Computational Fluid Dynamics (CFD) solvers. Algorithms must therefore minimize the number of samples required for learning (sample efficiency) and generate a usable policy from each training (reliability). This paper aims to (a) evaluate three existing algorithms (DDPG, TD3, and SAC) on a fluid mechanics problem with respect to reliability and sample efficiency across a range of training configurations, (b) establish a fluid mechanics benchmark of increasing data collection cost, and (c) provide practical guidelines and insights for the fluid dynamics practitioner. The benchmark consists in controlling an airfoil to reach a target. The problem is solved with either a low-cost low-order model or with a high-fidelity CFD approach. The study found that DDPG and TD3 have learning stability issues highly dependent on DRL hyperparameters and reward formulation, requiring therefore significant tuning. In contrast, SAC is shown to be both reliable and sample efficient across a wide range of parameter setups, making it well suited to solve fluid mechanics problems and set up new cases without tremendous effort. In particular, SAC is resistant to small replay buffers, which could be critical if full-flow fields were to be stored.

show abstract

Reinforcement-learning-based control of convectively unstable flows

Cited by 7 publications

References 101 publications

Active flow control for bluff body drag reduction using reinforcement learning with partial measurements

Active flow control for bluff body drag reduction using reinforcement learning with partial measurements

Recent advances in applying deep reinforcement learning for flow control: Perspectives and future directions

Reliability assessment of off-policy deep reinforcement learning: A benchmark for aerodynamics

Contact Info

Product

Resources

About