Speeding-Up Action Learning in a Social Robot With Dyna-Q+: A Bioinspired Probabilistic Model Approach

Abstract

Robotic systems developed for social and dynamic environments require adaptive mechanisms to successfully operate. In consequence, learning from rewards has provided meaningful results in applications involving human-robot interaction. Nonetheless, in those cases where the robot's state space and the number of actions is extensive, dimensionality becomes intractable, drastically slowing down the learning process. This effect is specially notorious in one-step temporal difference methods, as just one update is performed per robot-environment interaction. In this contribution, we prove how the action-based learning of a social robot can be improved by combining classical temporal difference reinforcement learning methods, such as Q-learning or Q(λ), with a probabilistic model of the environment. This architecture, called Dyna, allows the robot to simultaneously act and plan using the experience obtained during real human-robot interactions. Principally, Dyna improves classical algorithms in terms of convergence speed and stability, strengthening the learning process. Hence, in this work, a Dyna architecture is embedded in our social robot Mini, aiming at endowing it with the capability to autonomously maintain an optimal internal state while living in a dynamic environment.

A Systematic Literature Review of Decision-Making and Control Systems for Autonomous and Social Robots

“…The decision-making is grounded on the robot's motivations psychological states that represent the robot's needs. Then, three years later, in [85], they updated their previous model with Dyna-Q+, an RL algorithm that allows autonomous agents to speed up the learning process by representing a model of the environment.…”