Data-driven control of spatiotemporal chaos with reduced-order neural ODE-based models and reinforcement learning

Abstract

Deep reinforcement learning (RL) is a data-driven method capable of discovering complex control strategies for high-dimensional systems, making it promising for flow control applications. In particular, the present work is motivated by the goal of reducing energy dissipation in turbulent flows, and the example considered is the spatiotemporally chaotic dynamics of the Kuramoto-Sivashinsky equation (KSE). A major challenge associated with RL is that substantial training data must be generated by repeatedly interacting with the target system, making it costly when the system is computationally or experimentally expensive. We mitigate this challenge in a data-driven manner by combining dimensionality reduction via an autoencoder with a neural ODE framework to obtain a low-dimensional dynamical model from just a limited data set. We substitute this datadriven reduced-order model (ROM) in place of the true system during RL training to efficiently estimate the optimal policy, which can then be deployed on the true system. For the KSE actuated with localized forcing ("jets") at four locations, we demonstrate that we are able to learn a ROM that accurately captures the actuated dynamics as well as the underlying natural dynamics just from snapshots of the KSE experiencing random actuations. Using this ROM and a control objective of minimizing dissipation and power cost, we extract a control policy from it using deep RL. We show that the ROM-based control strategy translates well to the true KSE and highlight that the RL agent discovers and stabilizes an underlying forced equilibrium solution of the KSE system. We show that this forced equilibrium captured in the ROM and discovered through RL is related to an existing known equilibrium solution of the natural KSE.

Data-driven Control of Agent-based Models: an Equation/Variable-free Machine Learning Approach

“…Recent theoretical and technological advances in machine and deep learning have revolutionized the way we model, analyze and control high-dimensional multiscale/complex systems, ranging from complex fluids and materials [20,49,50] and process control [51,52] to the biological and biomedical sciences [53] and from environmental engineering and climate change [54] to sustainable mobility and robotics [55]. The high-dimensionality that intrinsically characterizes the state-space of relevant multiscale/complex problems, compounded by the inherent modelling uncertainties across scales, severely challenge our ability to efficiently understand, learn, analyze and control their collective behavior.…”

Learning a model is paramount for sample efficiency in reinforcement learning control of PDEs

“…This spatial configuration has been shown to manifest stable chaotic conditions, having the smallest system size to develop sustained (weak) turbulence [39]. Again following related work [34], [38], [40], the control consists of a superposition of several Gaussians…”

“…where u is the velocity and φ is an additive forcing term. Similar to other works [34], [38], we study a one-dimensional spatial domain Ω = [0, L] with periodic boundary conditions and system size L = 22. This spatial configuration has been shown to manifest stable chaotic conditions, having the smallest system size to develop sustained (weak) turbulence [39].…”

“…During the transient, the system is not actuated. In accordance with [34], [40], our goal is to dampen the dissipation D of the solution variable while minimizing the amount of energy P spent to power the system as well as the actuation devices:…”

“…Even though it seems clear that surrogates have the potential to improve the learning process, the trade off in terms of sample efficiency, computational complexity, and performance is seldom quantified, in particular in an engineering context such as PDE control. Two recent approaches using learned surrogates are [33] or [34]. In addition to assuming complete knowledge of the phenomena governing the system dynamics, the first work uses a physics-informed loss that, in essence, uses a deep neural network to compute the same finite difference solution as the numerical simulation does, but at a substantial slowdown in terms of runtime.…”