Online transfer learning strategy for enhancing the scalability and deployment of deep reinforcement learning control in smart buildings

Coraci, Davide; Brandi, Silvio; Hong, Tianzhen; Capozzoli, Alfonso

doi:10.1016/j.apenergy.2022.120598

Cited by 45 publications

(13 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where b occ,k is a boolean variable being 1 when occupants are present or 0 otherwise. A temperature violation T viol,k is calculated as the absolute temperature difference between the indoor temperature and the upper T i or lower limit T i , and can have different expressions depending on the value of the indoor temperature T i [9]:…”

Section: Methodsmentioning

confidence: 99%

Comparison of two deep reinforcement learning algorithms towards an optimal policy for smart building thermal control

Silvestri,

Coraci,

et al. 2023

J. Phys.: Conf. Ser.

Self Cite

View full text Add to dashboard Cite

Heating, Ventilation, and Air Conditioning (HVAC) systems are the main providers of occupant comfort, and at the same time, they represent a significant source of energy consumption. Improving their efficiency is essential for reducing the environmental impact of buildings. However, traditional rule-based and model-based strategies are often inefficient in real-world applications due to the complex building thermal dynamics and the influence of heterogeneous disturbances, such as unpredictable occupant behavior. In order to address this issue, the performance of two state-of-the-art model-free Deep Reinforcement Learning (DRL) algorithms, Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC), has been compared when the percentage valve opening is managed in a thermally activated building system, modeled in a simulated environment from data collected in an existing office building in Switzerland. Results show that PPO reduced energy costs by 18% and decreased temperature violations by 33%, while SAC achieved a 14% reduction in energy costs and 64% fewer temperature violations compared to the onsite Rule-Based Controller (RBC).

show abstract

Section: Methodsmentioning

confidence: 99%

Comparison of two deep reinforcement learning algorithms towards an optimal policy for smart building thermal control

Silvestri,

Coraci,

et al. 2023

J. Phys.: Conf. Ser.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The results suggested that transferring the DRL control policy from one building to another within the energy community yielded comparable performance while reducing the training costs. Coraci et al (2023b) developed an online transfer learning approach that exploits two knowledge-sharing techniques, weight-initialisation and IL, to transfer a DRL controller pre-trained on a source office building that minimises electricity cost while enhancing indoor temperature conditions by managing a cooling system. The proposed online transfer learning approach aims to replicate real-world implementation by simulating the transferred DRL agent in the target buildings for a single episode.…”

Section: Related Work On Tl Applications For Reinforcement Learning C...mentioning

confidence: 99%

“…Nevertheless, in the early stages of the training period, the agent possesses limited knowledge about the control problem, and there exists a significant risk that the chosen controller actions yield suboptimal performance. In this framework, the memory buffer of the online DRL agent is initialised with transitions acquired from the operation of the RBC, which is essentially an imitation learning approach (Coraci et al 2023b). The performance of the online DRL strategy depends strongly on the value of the number of gradient steps and learning rate.…”

Section: Performance Benchmarking Of Online Tl Strategy On Target Bui...mentioning

confidence: 99%

An innovative heterogeneous transfer learning framework to enhance the scalability of deep reinforcement learning controllers in buildings with integrated energy systems

Coraci,

Brandi,

Hong

et al. 2024

Build. Simul.

Self Cite

View full text Add to dashboard Cite

Deep Reinforcement Learning (DRL)-based control shows enhanced performance in the management of integrated energy systems when compared with Rule-Based Controllers (RBCs), but it still lacks scalability and generalisation due to the necessity of using tailored models for the training process. Transfer Learning (TL) is a potential solution to address this limitation. However, existing TL applications in building control have been mostly tested among buildings with similar features, not addressing the need to scale up advanced control in real-world scenarios with diverse energy systems. This paper assesses the performance of an online heterogeneous TL strategy, comparing it with RBC and offline and online DRL controllers in a simulation setup using EnergyPlus and Python. The study tests the transfer in both transductive and inductive settings of a DRL policy designed to manage a chiller coupled with a Thermal Energy Storage (TES). The control policy is pre-trained on a source building and transferred to various target buildings characterised by an integrated energy system including photovoltaic and battery energy storage systems, different building envelope features, occupancy schedule and boundary conditions (e.g., weather and price signal). The TL approach incorporates model slicing, imitation learning and fine-tuning to handle diverse state spaces and reward functions between source and target buildings. Results show that the proposed methodology leads to a reduction of 10% in electricity cost and between 10% and 40% in the mean value of the daily average temperature violation rate compared to RBC and online DRL controllers. Moreover, online TL maximises self-sufficiency and self-consumption by 9% and 11% with respect to RBC. Conversely, online TL achieves worse performance compared to offline DRL in either transductive or inductive settings. However, offline Deep Reinforcement Learning (DRL) agents should be trained at least for 15 episodes to reach the same level of performance as the online TL. Therefore, the proposed online TL methodology is effective, completely model-free and it can be directly implemented in real buildings with satisfying performance.

show abstract

“…Transfer Learning is emerging as a promising strategy to improve the wide-spread application of models. However, there are still significant research gaps regarding the identification of suitable training sources and the prediction of the performance after the transfer to another building [9,10].…”

Section: Motivationmentioning

confidence: 99%

Scalable decarbonisation using automated operation optimisation

Baranski,

Bode,

Nienaber

et al. 2023

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

One of the biggest challenges in facing the climate crisis is the decarbonization of the large and diverse building stock. A reduction of carbon dioxide emissions can be achieved by technical measures and engaging the building occupants to adapt their behaviour. Among the technical measures, implementing predictive control as an upgrade of the existing heating, ventilation, air conditioning and cooling system is especially promising as it allows reductions at potentially low running cost. However, the effort for adapting, implementing and deploying these methods to fit specific buildings and scenarios is high and requires special domain knowledge, hindering the wide-spread application. In this paper, we present a highly automated and data-driven implementation process utilizing an open-source container orchestration system, and the results from real-life case studies in existing buildings in which predictive control was retrofitted. Additionally, occupant information systems were installed in the buildings for increasing transparency about the building performance and the effect of the occupants’ behaviour. The shown method is useful for reducing the time required and manual effort for implementing new control strategies, and thus reducing carbon dioxide emissions while simultaneously increasing thermal comfort and air quality.

show abstract

Online transfer learning strategy for enhancing the scalability and deployment of deep reinforcement learning control in smart buildings

Cited by 45 publications

References 72 publications

Comparison of two deep reinforcement learning algorithms towards an optimal policy for smart building thermal control

Comparison of two deep reinforcement learning algorithms towards an optimal policy for smart building thermal control

An innovative heterogeneous transfer learning framework to enhance the scalability of deep reinforcement learning controllers in buildings with integrated energy systems

Scalable decarbonisation using automated operation optimisation

Contact Info

Product

Resources

About