Effective control of two-dimensional Rayleigh–Bénard convection: Invariant multi-agent reinforcement learning is all you need

Abstract

Rayleigh–Bénard convection (RBC) is a recurrent phenomenon in a number of industrial and geoscience flows and a well-studied system from a fundamental fluid-mechanics viewpoint. In the present work, we conduct numerical simulations to apply deep reinforcement learning (DRL) for controlling two-dimensional RBC using sensor-based feedback control. We show that effective RBC control can be obtained by leveraging invariant multi-agent reinforcement learning (MARL), which takes advantage of the locality and translational invariance inherent to RBC flows inside wide channels. MARL applied to RBC allows for an increase in the number of control segments without encountering the curse of dimensionality that would result from a naive increase in the DRL action-size dimension. This is made possible by the MARL ability for re-using the knowledge generated in different parts of the RBC domain. MARL is able to discover an advanced control strategy that destabilizes the spontaneous RBC double-cell pattern, changes the topology of RBC by coalescing adjacent convection cells, and actively controls the resulting coalesced cell to bring it to a new stable configuration. This modified flow configuration results in reduced convective heat transfer, which is beneficial in a number of industrial processes. We additionally draw comparisons with a conventional single-agent reinforcement learning (SARL) setup and report that in the same number of episodes, SARL is not able to learn an effective policy to control the cells. Thus, our work both shows the potential of MARL for controlling large RBC systems and demonstrates the possibility for DRL to discover strategies that move the RBC configuration between different topological configurations, yielding desirable heat-transfer characteristics.

Model-based deep reinforcement learning for accelerated learning from flow simulations

“…where the reference area is the projected area of all three cylinders in x-direction, namely A ref = 2.5dΔz . Definition (13) ensures that the sum over all cylinders recovers the correct total force coefficients:…”

“…Several remedies exist to reduce the computational cost and turnaround time, e.g., by exploiting invariances [12,13] or by pre-training on simulations with very coarse meshes [14]. While effective, the applicability of the aforementioned techniques strongly depends on the control problem, i.e., the problem must exhibit invariances, and the mesh coarsening must not change the flow characteristics.…”

Drag reduction in a minimal channel flow with scientific multi-agent reinforcement learning

“…Reinforcement learning (RL) has increasingly found applications in designing models and learning control policies in fluid mechanics [9]. It was introduced a decade ago for flow-control problems [10] and more recently it has been deployed in several fluiddynamics problems [11] and drag-reduction applications [12,13,14,15]. Other applications include fish bio-locomotion [16], optimization of aerial/aquatic vehicles' path and motion [17,18,19], active flow-control for bluff bodies [20,21,22], shape optimization [23], and learning closures [11] and wall models [24] for turbulent flows.…”

Deep reinforcement learning for active flow control in a turbulent separation bubble

“…where the main advances, tendencies, and types of problems addressed by DRL for AFC are discussed. Some of the most typical cases studied are drag reduction on a cylinder, both in 2D [22][23][24][25][26] and 3D [27][28][29]; convective heat reduction in Rayleigh-Bénard convection problems [30,31]; reduction of the skin-friction coefficient in turbulent channels [32,33]; and turbulence modeling [34][35][36]. Currently, the community is working towards expanding the use of DRL to higher complexity and more realistic cases.…”

“…A second approach, orthogonal to the multi-environment DRL, is to use a multi-agent reinforcement learning (MARL) method. First introduced by Belus et al [38] and later named by Vignon et al [31], the MARL method exploits the domain spatial invariants so that the dimensionality of the actuation space is reduced. For example, consider a set of multiple synthetic jets placed along the span of a circular cylinder.…”