A Deep Reinforcement Learning Approach to Improve the Learning Performance in Process Control

Abstract

Advanced model-based control methods have been widely used in industrial process control, but excellent performance requires regular maintenance of its model. Reinforcement learning can online update its policy through the observed data by interacting with the environment. Since a fast and stable learning process is required to improve the adaptability of the controller, we propose an improved deep deterministic actor critic predictor in this paper, where the immediate reward is separated from the action-value function to provide the actor with reliable gradient information at early stages. Then, an expectation form of policy gradient is developed based on the assumption that the state obeys the normal distribution. Simulation results show that the proposed algorithm achieves a more stable and faster learning procedure than those state-of-art deep reinforcement learning (DRL) algorithms. Meanwhile, the obtained policy achieves a more advantageous performance than the fine-tuned proportional integral and derivative (PID) and linear model predictive controllers, especially for those processes with nonlinearity. These indicate that the improved DRL controller has the potential to become an important tool in practical applications.

Deep Reinforcement Learning with Shallow Controllers: An Experimental Application to PID Tuning

“…In the model-based setting, Kim et al [29] incorporate deep neural networks (DNNs) as value function approximators into the globalized dual heuristic programming algorithm. Predictive models have also been augmented with popular DRL algorithms, such as DDPG or TD3, to improve the policy gradient estimation [30] Other approaches to RL-based control postulate a fixed control structure such as PID [31,32,33]. Brujeni et al [34] develop a model-free algorithm to dynamically select the PID gains from a pre-defined collection derived from Internal Model Control (IMC).…”

A survey and comparative evaluation of actor‐critic methods in process control

“…As to the use of application software, approximately 60% of the studies, utilized MATLAB software for controller development, integration, or real-time implementation. [60,63,69] Other researchers mostly used Tensorflow [64] and Python. [83] Some studies have integrated different functionalities of more than one software applications, for example, (i) MATLAB's process simulation module with Python's ANN generator APIs-PyTorch and Keras [70] and (ii) hybrid training module of Tensorflow and Python's Lambda deep learning workstation.…”

Meta-Reinforcement Learning for the Tuning of PI Controllers: An Offline Approach

“…This present work differs significantly from the approaches mentioned so far. Other approaches to more sample-efficient RL in process control utilize apprenticeship learning, transfer learning, or model-based strategies augmented with deep RL algorithms [29,21,30]. Our method differs in two significant ways: the training and deployment process is simplified with our meta-RL agent through its synthesized training over a large distribution of systems.…”

“…By picking a slow sampling time, the tank's dynamics appear faster from the perspective of the meta-RL agent. To geometrically center the time constant in Equation ( 28) to the meta-RL's task distribution, we set the sampling time to every 30 0.5 = 60 seconds. The true time constant of 55 seconds then appears as a time constant of 0.92 to the meta-RL agent.…”