Multi-Agent Tree Search with Dynamic Reward Shaping

Abstract

Sparse rewards and their representation in multi-agent domains remains a challenge for the development of multi-agent planning systems. While techniques from formal methods can be adopted to represent the underlying planning objectives, their use in facilitating and accelerating learning has witnessed limited attention in multi-agent settings. Reward shaping methods that leverage such formal representations in single-agent settings are typically static in the sense that the artificial rewards remain the same throughout the entire learning process. In contrast, we investigate the use of such formal objective representations to define novel reward shaping functions that capture the learned experience of the agents. More specifically, we leverage the automaton representation of the underlying team objectives in mixed cooperative-competitive domains such that each automaton transition is assigned an expected value proportional to the frequency with which it was observed in successful trajectories of past behavior. This form of dynamic reward shaping is proposed within a multi-agent tree search architecture wherein agents can simultaneously reason about the future behavior of other agents as well as their own future behavior.

Neurosymbolic Reinforcement Learning and Planning: A Survey

“…Transfer learning between the two different environments with the same objective is also eased with this approach. Authors in [69], extended the work by introducing Multiagent Tree Search Algorithm with reward shaping(MATS-A) so that it can be applied to multi-agent scenario and can handle both stochastic and deterministic transition in Multi-agent Non-Markovian Reward Decision Process. They prove that sharing the same search tree and DFA objective can be used to develop competitive and cooperative behavior among the agents, within and across the team.Research work [70] first converts an omega-regular specification into a Buchi automaton.…”

“…Neurosymbolic RL has been applied to different components of the RL framework, combining symbolic reasoning with neural networks to solve complex RL problems. It has been successful in addressing the issue of sparse rewards by formulating reward functions that provide more informative feedback to the agent [68], [69], [70]. It has also been used to learn programmatic policies that are more generalizable and flexible to different environments [71], [72], [73], [74].…”