Combining Reinforcement Learning and Causal Models for Robotics Applications

Abstract

The relation between Reinforcement learning (RL) and Causal Modeling(CM) is an underexplored area with untapped potential for any learning task. In this extended abstract of our Ph.D. research proposal, we present a way to combine both areas to improve their respective learning processes, especially in the context of our application area (service robotics). The preliminary results obtained so far are a good starting point for thinking about the success of our research project.

CARL: A Synergistic Framework for Causal Reinforcement Learning

“…The experimental results in the deterministic and stochastic versions of the test scenarios, show the advantages of CARL, not only over the previous work, but also compared to traditional model-free and modelbased algorithms. Fourth, the feasibility hypothesised in [11] of using the learned models for zero-shot transfer learning was verified using a more complex version of the Taxi task. Finally, the method's scalability to high-dimensional states, where the action-value function needs to be represented with deep neural networks, was verified.…”

“…Recently, in [11] the authors present a novel way to integrate reinforcement learning with causal discovery in the context of online MPD settings as the one described above, with an emphasis on reducing the policy learning time while inducing the underlying causal modes. In alternating stages, the proposed system learns a two-slice causal Dynamic Bayesian Network for each of the agent's actions and use those models to guide the action selection in the traditional RL episodes.…”

“…In this article, we extend [11] by proposing CARL, a causality-aware reinforcement learning framework which not only formalize the ideas but also improves and resolves several of the limitations of the previous work. First, a new stage (RL for CD) is introduced in order to obtain better data for causal discovery, that ensures that the structure of the learned causal models will be closer to the ground truth, compared to the traditional RL data collection process.…”

“…First, a new stage (RL for CD) is introduced in order to obtain better data for causal discovery, that ensures that the structure of the learned causal models will be closer to the ground truth, compared to the traditional RL data collection process. Second, the algorithm for using the learned models (RL using CD) proposed in [11] was improved to take into account not only the immediate high positive reward but also to filter the actions leading to a high negative reward. Third, an extensive series of experiments were carried out on the "Taxi task", a much more complex scenario than the "Coffee task" used in [11], where the robustness of CARL to parameters defined as critical in [11] was tested.…”

“…Second, the algorithm for using the learned models (RL using CD) proposed in [11] was improved to take into account not only the immediate high positive reward but also to filter the actions leading to a high negative reward. Third, an extensive series of experiments were carried out on the "Taxi task", a much more complex scenario than the "Coffee task" used in [11], where the robustness of CARL to parameters defined as critical in [11] was tested. The experimental results in the deterministic and stochastic versions of the test scenarios, show the advantages of CARL, not only over the previous work, but also compared to traditional model-free and modelbased algorithms.…”