Recent advances in applying deep reinforcement learning for flow control: Perspectives and future directions

Vignon, Colin; Rabault, Jean; Vinuesa, Ricardo

doi:10.1063/5.0143913

Cited by 65 publications

(19 citation statements)

References 166 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also, typically model-free RL requires large amounts of training data through interactions with the environment, which makes RL expensive and infeasible for certain applications. Further information about RL and its applications in fluid mechanics can be found in the reviews of Garnier et al (2021) and Vignon, Rabault & Vinuesa (2023).…”

Section: Model-free Active Flow Control By Reinforcement Learningmentioning

confidence: 99%

See 1 more Smart Citation

Active flow control for bluff body drag reduction using reinforcement learning with partial measurements

Xia,

Zhang,

Kerrigan

et al. 2024

J. Fluid Mech.

View full text Add to dashboard Cite

Active flow control for drag reduction with reinforcement learning (RL) is performed in the wake of a two-dimensional square bluff body at laminar regimes with vortex shedding. Controllers parametrised by neural networks are trained to drive two blowing and suction jets that manipulate the unsteady flow. The RL with full observability (sensors in the wake) discovers successfully a control policy that reduces the drag by suppressing the vortex shedding in the wake. However, a non-negligible performance degradation ( $\sim$ 50 % less drag reduction) is observed when the controller is trained with partial measurements (sensors on the body). To mitigate this effect, we propose an energy-efficient, dynamic, maximum entropy RL control scheme. First, an energy-efficiency-based reward function is proposed to optimise the energy consumption of the controller while maximising drag reduction. Second, the controller is trained with an augmented state consisting of both current and past measurements and actions, which can be formulated as a nonlinear autoregressive exogenous model, to alleviate the partial observability problem. Third, maximum entropy RL algorithms (soft actor critic and truncated quantile critics) that promote exploration and exploitation in a sample-efficient way are used, and discover near-optimal policies in the challenging case of partial measurements. Stabilisation of the vortex shedding is achieved in the near wake using only surface pressure measurements on the rear of the body, resulting in drag reduction similar to that in the case with wake sensors. The proposed approach opens new avenues for dynamic flow control using partial measurements for realistic configurations.

show abstract

Section: Model-free Active Flow Control By Reinforcement Learningmentioning

confidence: 99%

“…Further information about RL and its applications in fluid mechanics can be found in the reviews of Garnier et al. (2021) and Vignon, Rabault & Vinuesa (2023).…”

Section: Introductionmentioning

confidence: 99%

Active flow control for bluff body drag reduction using reinforcement learning with partial measurements

Xia,

Zhang,

Kerrigan

et al. 2024

J. Fluid Mech.

View full text Add to dashboard Cite

show abstract

“…An extensive account of the potential for flow control from DRL, including turbulent wings, can be found in Refs. [65,68].…”

Section: Control Of Turbulencementioning

confidence: 99%

Modelling and controlling turbulent flows through deep learning

Vinuesa

2022

Preprint

View full text Add to dashboard Cite

The advent of new powerful deep neural networks (DNNs) has fostered their application in a wide range of research areas, including more recently in fluid mechanics. In this presentation, we will cover some of the fundamentals of deep learning applied to computational fluid dynamics (CFD). Furthermore, we explore the capabilities of DNNs to perform various predictions in turbulent flows: we will use convolutional neural networks (CNNs) for non-intrusive sensing, i.e. to predict the flow in a turbulent open channel based on quantities measured at the wall. We show that it is possible to obtain very good flow predictions, outperforming traditional linear models, and we showcase the potential of transfer learning between friction Reynolds numbers of 180 and 550. We also discuss other modelling methods based on autoencoders (AEs) and generative adversarial networks (GANs), and we present results of deep-reinforcement-learning-based flow control.

show abstract

“…The reader is referred to Ref. [19] for a wide overview of DRL for AFC, where the most representative works of different problems are discussed. Some of the most typical cases studied are drag reduction on a cylinder, both in 2D [20,21,22,23,24] and 3D [25,26,27]; convective heat reduction in Rayleigh-Bénard problems [28,29]; reduction of the skin-friction coefficient in turbulent channels [30,31]; and turbulence modelling [32,33,34].…”

Section: Introductionmentioning

confidence: 99%

“…[19] for a wide overview of DRL for AFC, where the most representative works of different problems are discussed. Some of the most typical cases studied are drag reduction on a cylinder, both in 2D [20,21,22,23,24] and 3D [25,26,27]; convective heat reduction in Rayleigh-Bénard problems [28,29]; reduction of the skin-friction coefficient in turbulent channels [30,31]; and turbulence modelling [32,33,34]. The increasing tendency of tackling more computationally expensive cases has motivated the development of novel DRL techniques that can accelerate the training process of a control strategy.…”

Section: Introductionmentioning

confidence: 99%

Active flow control of a turbulent separation bubble through deep reinforcement learning

Font,

Alcántara-Ávila,

Rabault

et al. 2024

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

The control efficacy of classical periodic forcing and deep reinforcement learning (DRL) is assessed for a turbulent separation bubble (TSB) at Reτ = 180 on the upstream region before separation occurs. The TSB can resemble a separation phenomenon naturally arising in wings, and a successful reduction of the TSB can have practical implications in the reduction of the aviation carbon footprint. We find that the classical zero-net-mas-flux (ZNMF) periodic control is able to reduce the TSB by 15.7%. On the other hand, the DRL-based control achieves 25.3% reduction and provides a smoother control strategy while also being ZNMF. To the best of our knowledge, the current test case is the highest Reynolds-number flow that has been successfully controlled using DRL to this date. In future work, these results will be scaled to well-resolved large-eddy simulation grids. Furthermore, we provide details of our open-source CFD–DRL framework suited for the next generation of exascale computing machines.

show abstract

Recent advances in applying deep reinforcement learning for flow control: Perspectives and future directions

Cited by 65 publications

References 166 publications

Active flow control for bluff body drag reduction using reinforcement learning with partial measurements

Active flow control for bluff body drag reduction using reinforcement learning with partial measurements

Modelling and controlling turbulent flows through deep learning

Active flow control of a turbulent separation bubble through deep reinforcement learning

Contact Info

Product

Resources

About