Comparison of Deep Reinforcement Learning and PID Controllers for Automatic Cold Shutdown Operation

Abstract

Many industries apply traditional controllers to automate manual control. In recent years, artificial intelligence controllers applied with deep-learning techniques have been suggested as advanced controllers that can achieve goals from many industrial domains, such as humans. Deep reinforcement learning (DRL) is a powerful method for these controllers to learn how to achieve their specific operational goals. As DRL controllers learn through sampling from a target system, they can overcome the limitations of traditional controllers, such as proportional-integral-derivative (PID) controllers. In nuclear power plants (NPPs), automatic systems can manage components during full-power operation. In contrast, startup and shutdown operations are less automated and are typically performed by operators. This study suggests DRL-based and PID-based controllers for cold shutdown operations, which are a part of startup operations. By comparing the suggested controllers, this study aims to verify that learning-based controllers can overcome the limitations of traditional controllers and achieve operational goals with minimal manipulation. First, to identify the required components, operational goals, and inputs/outputs of operations, this study analyzed the general operating procedures for cold shutdown operations. Then, PID- and DRL-based controllers are designed. The PID-based controller consists of PID controllers that are well-tuned using the Ziegler–Nichols rule. The DRL-based controller with long short-term memory (LSTM) is trained with a soft actor-critic algorithm that can reduce the training time by using distributed prioritized experience replay and distributed learning. The LSTM can process a plant time-series data to generate control signals. Subsequently, the suggested controllers were validated using an NPP simulator during the cold shutdown operation. Finally, this study discusses the operational performance by comparing PID- and DRL-based controllers.

An overview of power reactor kinetics and control in load-following operation modes

“…Lee et al (2020) describes a prior DRL approach for control automation of a later phase of the NPP startup, for increasing the reactor power from 2% to 100% at a specified rate of power increase. Lee et al (2022) discussed a DRL approach for automatic cold shutdown operation.…”

Dynamic Reward in DQN for Autonomous Navigation of UAVs Using Object Detection

“…To enable a UAV to execute a multitude of complex tasks in a 3D space, an approach could be to use lowlevel flight control coupled with a supervisory rule-based or fuzzy controller to optimize the drone's navigation [10]. However, traditional controllers, such as PID, are not effective in handling dynamic changes such as wind gusts or turbulence, which can significantly affect the UAV's flight trajectory [11]. Optimizing UAV's trajectories is one of the aims of automating inspections, with the goal of enhancing its efficiency.…”