Research and Application of Predictive Control Method Based on Deep Reinforcement Learning for HVAC Systems

Fu, Chenhui; Zhang, Yunhua

doi:10.1109/access.2021.3114161

Cited by 14 publications

(7 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The results showed that the predictive control method could ensure thermal comfort and air quality in the indoor environment while minimizing energy consumption. Fu et al [35] established a thermal dynamics model to predict the future trend of HVAC systems. Twin delayed deep deterministic policy gradient algorithm and model predictive control (TD3-MPC) were proposed to pre-adjust building temperatures at off-peak times.…”

Section: Predictive Controlmentioning

confidence: 99%

Building Energy Consumption Prediction Using a Deep-Forest-Based DQN Method

Chen

et al. 2022

Buildings

View full text Add to dashboard Cite

When deep reinforcement learning (DRL) methods are applied in energy consumption prediction, performance is usually improved at the cost of the increasing computation time. Specifically, the deep deterministic policy gradient (DDPG) method can achieve higher prediction accuracy than deep Q-network (DQN), but it requires more computing resources and computation time. In this paper, we proposed a deep-forest-based DQN (DF–DQN) method, which can obtain higher prediction accuracy than DDPG and take less computation time than DQN. Firstly, the original action space is replaced with the shrunken action space to efficiently find the optimal action. Secondly, deep forest (DF) is introduced to map the shrunken action space to a single sub-action space. This process can determine the specific meaning of each action in the shrunken action space to ensure the convergence of DF–DQN. Thirdly, state class probabilities obtained by DF are employed to construct new states by considering the probabilistic process of shrinking the original action space. The experimental results show that the DF–DQN method with 15 state classes outperforms other methods and takes less computation time than DRL methods. MAE, MAPE, and RMSE are decreased by 5.5%, 7.3%, and 8.9% respectively, and R2 is increased by 0.3% compared to the DDPG method.

show abstract

Section: Predictive Controlmentioning

confidence: 99%

Building Energy Consumption Prediction Using a Deep-Forest-Based DQN Method

Chen

et al. 2022

Buildings

View full text Add to dashboard Cite

show abstract

“…In such cases, AC methods were utilized to tune controller parameters, [57,58] or identify the process dynamics [59] and help generate control actions. [60][61][62][63] Overall, AC methods have been found very suitable for set-point tracking and disturbance rejection in real-time implementations. Table 1 summarizes important studies, which utilized AC methods for process control.…”

Section: Process Control Applicationsmentioning

confidence: 99%

“…As to the use of application software, approximately 60% of the studies, utilized MATLAB software for controller development, integration, or real-time implementation. [60,63,69] Other researchers mostly used Tensorflow [64] and Python. [83] Some studies have integrated different functionalities of more than one software applications, for example, (i) MATLAB's process simulation module with Python's ANN generator APIs-PyTorch and Keras [70] and (ii) hybrid training module of Tensorflow and Python's Lambda deep learning workstation.…”

Section: Process Control Applicationsmentioning

confidence: 99%

A survey and comparative evaluation of actor‐critic methods in process control

Dutta

Upreti

2022

Can J Chem Eng

View full text Add to dashboard Cite

Actor-critic (AC) methods have emerged as an important class of reinforcement learning (RL) paradigm that enables model-free control by acting on a process and learning from the consequence. To that end, these methods utilize artificial neural networks, which are synergized for action evaluation and optimal action prediction. This feature is highly desirable for process control, especially when the knowledge about a process is limited or when it is susceptible to uncertainties. In this work, we summarize important concepts of AC methods and survey their process control applications. This treatment is followed by a comparative evaluation of the set-point tracking and robustness of controllers based on five prominent AC methods, namely, DDPG, TD3, SAC, PPO, and TRPO, in five case studies of varying process nonlinearity. The training demands and control performances indicate the superiority of DDPG and TD3 methods, which rely on off-policy, deterministic search for optimal action policies. Overall, the knowledge base and results of this work are expected to serve practitioners in their efforts toward further development of autonomous process control strategies.

show abstract

“…However, to ensure an accurate simulation, some authors run the simulator with a shorter timestep, e.g., 5 min, in which case the simulator takes several steps adding up to 15 min before the next interaction with the RL agent is performed. Some works use longer control timesteps such as 20 min [78] or 60 min [79]. Depending on how the environment has been implemented, and how long it takes for the system to stabilize after a control action, a shorter control time step could result in unstable feedback to the RL agent [80].…”

Section: Overview Of the Analyzed Articlesmentioning

confidence: 99%

A Review of Reinforcement Learning Applications to Control of Heating, Ventilation and Air Conditioning Systems

2022

View full text Add to dashboard Cite

Reinforcement learning has emerged as a potentially disruptive technology for control and optimization of HVAC systems. A reinforcement learning agent takes actions, which can be direct HVAC actuator commands or setpoints for control loops in building automation systems. The actions are taken to optimize one or more targets, such as indoor air quality, energy consumption and energy cost. The agent receives feedback from the HVAC systems to quantify how well these targets have been achieved. The feedback is captured by a reward function designed by the developer of the reinforcement learning agent. A few reviews have focused on the reward aspect of reinforcement learning applications for HVAC. However, there is a lack of reviews that assess how the actions of the reinforcement learning agent have been formulated, and how this impacts the possibilities to achieve various optimization targets in single zone or multi-zone buildings. The aim of this review is to identify the action formulations in the literature and to assess how the choice of formulation impacts the level of abstraction at which the HVAC systems are considered. Our methodology involves a search string in the Web of Science database and a list of selection criteria applied to each article in the search results. For each selected article, a three-tier categorization of the selected articles has been performed. Firstly, the applicability of the approach to buildings with one or more zones is considered. Secondly, the articles are categorized by the type of action taken by the agent, such as a binary, discrete or continuous action. Thirdly, the articles are categorized by the aspects of the indoor environment being controlled, namely temperature, humidity or air quality. The main result of the review is this three-tier categorization that reveals the community’s emphasis on specific HVAC applications, as well as the readiness to interface the reinforcement learning solutions to HVAC systems. The article concludes with a discussion of trends in the field as well as challenges that require further research.

show abstract

Research and Application of Predictive Control Method Based on Deep Reinforcement Learning for HVAC Systems

Cited by 14 publications

References 28 publications

Building Energy Consumption Prediction Using a Deep-Forest-Based DQN Method

Building Energy Consumption Prediction Using a Deep-Forest-Based DQN Method

A survey and comparative evaluation of actor‐critic methods in process control

A Review of Reinforcement Learning Applications to Control of Heating, Ventilation and Air Conditioning Systems

Contact Info

Product

Resources

About