A practically implementable reinforcement learning‐based process controller design

Abstract

The present article enables reinforcement learning (RL)‐based controllers for process control applications. Existing instances of RL‐based solutions have significant challenges for online implementation since the training process of an RL agent (controller) presently requires practically impossible number of online interactions between the agent and the environment (process). To address this challenge, we propose an implementable model‐free RL method developed by leveraging industrially implemented model predictive control (MPC) calculations (often designed using a simple linear model identified via step tests). In the first step, MPC calculations are used to pretrain an RL agent that can mimic the MPC performance. Specifically, the MPC calculations are used to pretrain the actor, and the objective function is used to pretrain the critic(s). The pretrained RL agent is then employed within a model‐free RL framework to control the process in a way that initially imitates MPC behavior (thus not compromising process performance and safety), but also continuously learns and improve its performance over the nominal linear MPC. The effectiveness of the proposed approach is illustrated through simulations on a chemical reactor example.

Gas-Lift Optimization Using Physics-Informed Deep Reinforcement Learning

“…This is similar to the methodologies developed by Delou et al, Curvelo et al and Demuner et al, which employed a strategy centered on a Hammerstein model structure in which an identified process steady-state model represents its static nonlinear function. Our approach presents similarities to the methodologies developed in Hassanpour et al, , which created a reinforcement learning-based process controller design that can be practically implemented. They trained the DRL agent offline using MPC information through process demonstrations, and the actor and critic policies and the buffer were updated in real time.…”