Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms on a Building Energy Demand Coordination Task

Dhamankar, Gauraang; Vázquez-Canteli, José R.; Nagy, Zoltán

doi:10.1145/3427773.3427870

Cited by 10 publications

(16 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To compare our results with those of previous publications, we made sure that the scenarios and scenario parameters matched those of Papoudakis et al [5] and Atrazhev et al [19], and the results were compared to the results of those previous works.…”

Section: Methodsmentioning

confidence: 99%

“…To remain consistent with previous publications, the LBF scenarios selected for this study are 8x8-2p-2f-coop, 2s-8x8-2p-2f-coop, 10x10-3p-3f, and 2s-10x10-3p-3f. Algorithms are also selected based on these criteria: IQL [16], IA2C [6], IPPO [17], MAA2C [5], MAPPO [7], VDN [10] and QMIX [9] were selected as they are studied in both Papoudakis et al [5] and in Atrazhev et al [19] and represent an acceptable assortment of independent algorithms, centralized critic CLDE algorithms, and value factorization CLDE algorithms.…”

Section: Methodsmentioning

confidence: 99%

“…During execution, the centralized element is removed, and the agent's policy is based on conditions only on local observations. This has been shown to increase the coordination of agents [5]. CLDE algorithms separate into two major categories: centralized policy gradient methods [6][7][8] and value decomposition methods [9,10].…”

Section: Introductionmentioning

confidence: 99%

“…We investigate how different CLDE algorithm performances change as a result of this change and discuss this performance change. In this work, we study the effect of varying reward functions from joint rewards to individual rewards on Independent Q Learning (IQL) [16], Independent Proximal Policy Optimization (IPPO) [17], independent synchronous actor-critic (IA2C) [6], multi-agent proximal policy optimization (MAPPO) [7], multi agent synchronous actorcritic (MAA2C) [5,6], value decomposition networks (VDN) [10], and QMIX [9] when evaluated on the LBF environment [18]. This environment was chosen as it is a gridworld environment, and therefore simpler to understand when compared to other MARL environments such as those based on the StarCraft environment; however, it is a very challenging environment that requires cooperation to solve and has the ability to include the forcing of cooperative policies and partial observability for study.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

It’s All about Reward: Contrasting Joint Rewards and Individual Reward in Centralized Learning Decentralized Execution Algorithms

Atrazhev

Musilek

2023

Systems

View full text Add to dashboard Cite

This paper addresses the issue of choosing an appropriate reward function in multi-agent reinforcement learning. The traditional approach of using joint rewards for team performance is questioned due to a lack of theoretical backing. The authors explore the impact of changing the reward function from joint to individual on learning centralized decentralized execution algorithms in a Level-Based Foraging environment. Empirical results reveal that individual rewards contain more variance, but may have less bias compared to joint rewards. The findings show that different algorithms are affected differently, with value factorization methods and PPO-based methods taking advantage of the increased variance to achieve better performance. This study sheds light on the importance of considering the choice of a reward function and its impact on multi-agent reinforcement learning systems.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

It’s All about Reward: Contrasting Joint Rewards and Individual Reward in Centralized Learning Decentralized Execution Algorithms

Atrazhev

Musilek

2023

Systems

View full text Add to dashboard Cite

show abstract

“…The CityLearn environment is an OpenAI environment which allows the control of domestic hot water and chilled water storage in a district environment. This environment was also used by Dhamankar et al [31] who compared three classes of multi-agent RL algorithms for demand response and coordination of a district. Note that for this environ-ment, the conditioning loads were precomputed for a given set of climates.…”

Section: Action a T Reward R T+1mentioning

confidence: 99%

Development of a Soft Actor Critic Deep Reinforcement Learning Approach for Harnessing Energy Flexibility in a Large Office Building

Kathirgamanathan,

Mangina,

Finn

2021

Preprint

View full text Add to dashboard Cite

This research is concerned with the novel application and investigation of 'Soft Actor Critic' (SAC) based Deep Reinforcement Learning (DRL) to control the cooling setpoint (and hence cooling loads) of a large commercial building to harness energy flexibility. The research is motivated by the challenge associated with the development and application of conventional model-based control approaches at scale to the wider building stock. SAC is a model-free DRL technique that is able to handle continuous action spaces and which has seen limited application to real-life or high-fidelity simulation implementations in the context of automated and intelligent control of building energy systems. Such control techniques are seen as one possible solution to supporting the operation of a smart, sustainable and future electrical grid. This research tests the suitability of the SAC DRL technique through training and deployment of the agent on an EnergyPlus based environment of the office building. The SAC DRL was found to learn an optimal control policy that was able to minimise energy costs by 9.7% compared to the default rule-based control (RBC) scheme and was able to improve or maintain thermal comfort limits over a test period of one week. The algorithm was shown to be robust to the different hyperparameters and this optimal control policy was learnt through the use of a minimal state space consisting of readily available variables. The robustness of the algorithm was tested through investigation of the speed of learning and ability to deploy to different seasons and climates. It was found that the SAC DRL requires minimal training sample points and outperforms the RBC after three months of operation and also without disruption to thermal comfort during this period. The agent is transferable to other climates and seasons although further retraining or hyperparameter tuning is recommended. Highlights:• A novel application of Soft Actor Critic Deep Reinforcement Learning (SAC DRL).• Controller harnesses energy flexibility from building passive thermal mass.• A novel investigation into robustness of hyperparameters and state space design.

show abstract

Reducing Peak Electricity Demands of a Cluster of Buildings with Multi-Agent Reinforcement Learning

Balwant

Basa

Misra

2023

Springer Proceedings in Mathematics &Amp; Statistics

View full text Add to dashboard Cite

Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms on a Building Energy Demand Coordination Task

Cited by 10 publications

References 3 publications

It’s All about Reward: Contrasting Joint Rewards and Individual Reward in Centralized Learning Decentralized Execution Algorithms

It’s All about Reward: Contrasting Joint Rewards and Individual Reward in Centralized Learning Decentralized Execution Algorithms

Development of a Soft Actor Critic Deep Reinforcement Learning Approach for Harnessing Energy Flexibility in a Large Office Building

Reducing Peak Electricity Demands of a Cluster of Buildings with Multi-Agent Reinforcement Learning

Contact Info

Product

Resources

About