Deep Integration of Physical Humanoid Control and Crowd Navigation

Abstract

Many multi-agent navigation approaches make use of simplified representations such as a disk. These simplifications allow for fast simulation of thousands of agents but limit the simulation accuracy and fidelity. In this paper, we propose a fully integrated physical character control and multi-agent navigation method. In place of sample complex online planning methods, we extend the use of recent deep reinforcement learning techniques. This extension improves on multi-agent navigation models and simulated humanoids by combining Multi-Agent and Hierarchical Reinforcement Learning. We train a single short term goal-conditioned low-level policy to provide directed walking behaviour. This task-agnostic controller can be shared by higher-level policies that perform longer-term planning. The proposed approach produces reciprocal collision avoidance, robust navigation, and emergent crowd behaviours. Furthermore, it offers several key affordances not previously possible in multi-agent navigation including tunable character morphology and physically accurate interactions with agents and the environment. Our results show that the proposed method outperforms prior methods across environments and tasks, as well as, performing well in terms of zero-shot generalization over different numbers of agents and computation time.

A Survey on Reinforcement Learning Methods in Character Animation

“…Haworth et al [HBM*20] introduces a method on the borderline between single character and crowd animation. By employing a method based on the ideas of Hierarchical Reinforcement Learning, they train two policies interacting with one another.…”

Path planning for multiple agents in an unknown environment using soft actor critic and curriculum learning

“…Kostrikov 26 identified an implicit bias in the algorithms based on the adversarial imitation learning framework and proposed a new algorithm called discriminator-actor-critic. Haworth 27 trained a low-level policy for walking behavior and a high-level policy for planning by hierarchical reinforcement learning. 28 Curriculum learning was first proposed by Bengio, 29 which is a training strategy 30 that trains the machine learning model from easy data subsets to hard dataset until the whole training dataset.…”

Crowd navigation in an unknown and complex environment based on deep reinforcement learning

“…We solve the above problems from two aspects. First, based on the idea of hierarchical reinforcement learning, 19 the task of the agent moving from the starting point to the goal is decomposed into a series of subtasks by extracting the key navigation points, 20 and then each subtask is planned in a divide and conquer manner. Therefore, how to determine the key navigation points is very important.…”