Exploiting locality and translational invariance to design effective deep reinforcement learning control of the 1-dimensional unstable falling liquid film

Abstract

Instabilities arise in a number of flow configurations. One such manifestation is the development of interfacial waves in multiphase flows, such as those observed in the falling liquid film problem. Controlling the development of such instabilities is a problem of both academic and industrial interest. However, this has proven challenging in most cases due to the strong nonlinearity and high dimensionality of the underlying equations. In the present work, we successfully apply Deep Reinforcement Learning (DRL) for the control of the one-dimensional (1D) depth-integrated falling liquid film. In addition, we introduce for the first time translational invariance in the architecture of the DRL agent, and we exploit locality of the control problem to define a dense reward function. This allows to both speed up learning considerably, and to easily control an arbitrary large number of jets and overcome the curse of dimensionality on the control output size that would take place using a naive approach. This illustrates the importance of the architecture of the agent for successful DRL control, and we believe this will be an important element in the effective application of DRL to large two-dimensional (2D) or three-dimensional (3D) systems featuring translational, axisymmetric or other invariants. :1910.07788v1 [physics.flu-dyn] arXiv

Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning

“…Although the computational cost remains a challenge to the wide application of DRL within fluid mechanics, this challenge can be progressively solved owing to the rapid advancement of high-performance computing architectures. Therefore, it is anticipated that significantly more complex problems, such as instabilities in boundary layers, 69,70 can be tackled using methodologies based on the present work, possibly in combination with other results and technical improvements such as the encoding of physical invariance of the system to control within the ANN architecture, 71,72 or the identification of reduced-order, hidden features of these systems. 73,74 In order to support the further development of DRL applications in the fluid mechanics community, all codes used are released as an open source (see Appendix A).…”

Stabilization of the fluidic pinball with gradient-enriched machine learning control

“…The inclusion of known symmetries or invariants in gMLC might be achieved by pre-testing and excluding individuals which strongly depart from these constraints. An example of self-discovery of such symmetries and invariants is reported in Belus et al (2019) for deep reinforcement learning.…”

Robust flow control and optimal sensor placement using deep reinforcement learning

“…Belus et al. (2019) leveraged spatial invariance to replicate identical DRL-learned sensor–actuator systems along a one-dimensional liquid film to control its instability. These simple yet nonlinear test cases serve as proof of concept and illustrate the promising potential of DRL for real-world flow control cases.…”