Makf-Sr: Multi-Agent Adaptive Kalman Filtering-Based Successor Representations

Abstract

The paper is motivated by the importance of the Smart Cities (SC) concept for future management of global urbanization and energy consumption. Multi-agent Reinforcement Learning (RL) is an efficient solution to utilize large amount of sensory data provided by the Internet of Things (IoT) infrastructure of the SCs for city-wide decision making and managing demand response. Conventional Model-Free (MF) and Model-Based (MB) RL algorithms, however, use a fixed reward model to learn the value function rendering their application challenging for ever changing SC environments. Successor Representations (SR)-based techniques are attractive alternatives that address this issue by learning the expected discounted future state occupancy, referred to as the SR, and the immediate reward of each state. SR-based approaches are, however, mainly developed for single agent scenarios and have not yet been extended to multi-agent settings. The paper addresses this gap and proposes the Multi-Agent Adaptive Kalman Filtering-based Successor Representation (MAKF-SR) framework. The proposed framework can adapt quickly to the changes in a multi-agent environment faster than the MF methods and with a lower computational cost compared to MB algorithms. The proposed MAKF-SR is evaluated through a comprehensive set of experiments illustrating superior performance compared to its counterparts.

Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning☆

“…An interesting perspective is to transfer knowledge across tasks with different action and state spaces (thus, different feature vectors). Furthermore, multi-agent RL [121,135] is an essential sub-field of RL. Extending existing transfer learning techniques to the multi-agent scenario is challenging because multi-agent RL typically necessitates socially desirable behaviors [136].…”

“…SU [40] and RaSF [119] estimated uncertainty within the SF-based transfer learning domain using Bayesian linear regressions and optimizing entropic utilities, respectively. Geerts et al [120] and Salimibeni et al [121] incorporated KFs into TD-SF frameworks to derive the uncertainty of the SF within their SF-based transfer learning algorithms for finite state spaces and multiple agents problems, respectively. Recently, Malekzadeh et al [86] combined multiple-model adaptive estimation with the TD-SF method to estimate the uncertainty of the approximated SF.…”

AKF-SR: Adaptive Kalman filtering-based successor representation

“…Less sensitivity to parameter settings improves the reproducibility aspect of a reliable algorithm to regenerate more consistent performances across multiple learning runs; therefore, reduces the risk of generating unpredictable performances in different practical applications [38]. Geerts et al [39] and Salimibeni et al [40] applied KTD framework for the SR estimation in RL problems, respectively, with discrete state spaces and multiple agents. The proposed algorithms, however, do not use the uncertainty information of the estimated SR, which can be achieved from KTD algorithm, for the action selection process.…”

Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation

“…Generally speaking, the main underlying objective is learning (via trial and error) from previous interactions of an autonomous agent and its surrounding environment. The optimal control (action) policy can be obtained via RL algorithms through the feedback that environment provides to the agent after each of its actions [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ]. Policy optimality can be reached via such an approach with the goal of increasing the reward over time.…”